Comments for Blog of Data

Comment on Tracking down the number of Firefox Addon users with hadoop by peter

peter — Fri, 28 Aug 2009 18:54:26 +0000

A prime example for the usage of hadoop. Great!
But I don’t think, that the conclusions you gained are reliable. Maybe FF should introduce an anonymous ID for each program, so the issues with multiple users behind an IP or changing ones are solved.
But even then, we can’t be sure, about the spread of particular addons, since update pings maybe skipped a day (People should take a walk on weekends and not browse the internet… ;o) So there wouldn’t be an update check right?)

Comment on Tracking down the number of Firefox Addon users with hadoop by deinspanjer

deinspanjer — Wed, 19 Aug 2009 16:03:49 +0000

These “versioncheck” pings normally happen on a 24 hour timer. There is a slight complication though in that when a user is updating Firefox itself, there are a couple of additional add-on versioncheck pings to determine compatibility with the Firefox version being installed, and if add-ons are updated, then the new versions of them ping again.

Comment on Tracking down the number of Firefox Addon users with hadoop by SteveL

SteveL — Wed, 19 Aug 2009 13:37:21 +0000

We in the Hadoop project are always pleased to see other OSS projects doing interesting stuff with our code, thank you for the writeup. And no, that hadoop on ubuntu wiki page is still lagging -we should really do our own .deb files and so have much simpler setup instructions: “select Apache Hadoop on Synaptic”.

A big issue for the stats is “how often are updates checked for”; without data on that, you can’t be sure what the numbers really mean -interesting as they are.

Comment on Tracking down the number of Firefox Addon users with hadoop by city_zen

city_zen — Mon, 17 Aug 2009 05:09:45 +0000

Thanks for publishing this.
I’m surprised that the number of users with add-ons installed is not HIGHER. I mean, about 50% of Firefox users don’t have EVEN ONE add-on installed? Hard to believe. I think I would use another browser if Firefox didn’t have add-ons to customize it. Don’t take me wrong, Firefox is a great browser, but for me it’s the add-ons that push it over any other browser out there.
Would it be possible to publish some more information about the usage patterns that you found? Starting with 28 GB of data, there must be more than 3 or 4 bullet points worth of output info, right? ;)
A few examples that maybe are available to you after all that processing (great job, btw):
- Top ten most installed add-ons
- Percentage of users with more than 10 add-ons installed
- Maximum number of add-ons installed (don’t look at me, I ONLY have a few dozens installed :D )
- Distribution by country
- OS used
etc.

Thank you

Comment on Update: Bugzilla SQR by deinspanjer

deinspanjer — Sat, 15 Aug 2009 16:32:14 +0000

http://lmgtfy.com/?q=ETL
SQR is a tough one. It stands for Software Quality Reports. It was the original name for the project that allows us to generate useful reports on Bugzilla data.
Here you go. ;)

Comment on Update: Bugzilla SQR by Mossop

Mossop — Sat, 15 Aug 2009 09:27:07 +0000

SQL, ETL, all sounds very exciting, if only there was some explanation of what it means.

Comment on Tracking down the number of Firefox Addon users with hadoop by skrueger

skrueger — Wed, 12 Aug 2009 18:21:53 +0000

Ingo: I see how frequently changing IPs is an issue, and I did not think of this problem while developing. But I don’t think this is a large problem because I only think that this could make our distinct IP smaller than it actually is since the FIrefox Add-on Manager will ping once in a 24hour time window. The problem I am thinking of could happen when user A from IP x.x.x.x pings for updates and then loses/changes his IP. Then user B picks up IP x.x.x.x and pings for updates. In this case 2 distinct users (not behind the same router) pinged for updates but only one IP was counted. However, I am not certain on how often this case happens.

Comment on Tracking down the number of Firefox Addon users with hadoop by Ingo Lütkebohle

Ingo Lütkebohle — Wed, 12 Aug 2009 09:57:41 +0000

Thanks for writing up your modeling approach, that is very good example material for using hadoop!

I have a question about your ip-to-user estimate: At least in Germany, many users do not keep an IP address for more than 24 hours. This is because almost all consumer broadband/DSL providers disconnect them about once a day. Routers immediately reconnect, but get assigned a different IP then. Do you think this is of relevance for the number of users per IP and if yes, have you accounted for it?

Comment on Tracking down the number of Firefox Addon users with hadoop by Wladimir Palant

Wladimir Palant — Mon, 10 Aug 2009 23:02:36 +0000

Interesting results. This seems to suggest that roughly 50% of all Firefox users have add-ons installed. If we then take 36 million users pinging AMO on a day (at least that’s how I interpret your numbers) and 8730182 pings for Adblock Plus on that day – we get that 12% of all Firefox users are using Adblock Plus. Finally a way to put ADU numbers in relation, the result is significantly higher than my previous estimates however. One possible explanation is that users who have add-ons installed typically use their browser more frequently and send out more pings than those who don’t. This would make it difficult to compare ADU numbers between Firefox and add-ons, add-on users are probably significantly fewer than 50%.

Comment on Shell script analytics by Arthur

Arthur — Thu, 30 Jul 2009 07:49:12 +0000

@Jesse: ~75% Intel vs. ~25% PPC