Skip to content

Mozilla, Firefox, “fixing computers” & a room full of kindergartners

20-May-09

My son has a vague concept of what I do at work. He knows I work at “Firefox”, knows the dino and knows I “fix computers”.

He knows that if he wants to get online he has to double-click on the Firefox icon and then on the Kidzui icon:kidzui-toolbar

He asked me if I could come to his school and show people how I “fix computers”.

I started off by asking if anyone recognized the logo on the back of my shirt.  Even to a room full of 5 year olds, the Firefox logo was instantly recognizable – there wasn’t anyone who didn’t know what it was.  I talked briefly about what Mozilla did (“we make a web browser”) and that I help fix computers when they break.  

I thought about showing them Mitchell’s Mozilla Tree but probably couldn’t have done as well as Mitchell could have! 

Then I pulled out my laptop and showed them Firefox (Minefield really).  I was in the middle of showing them Firefox and how what my son does when he wants to get on the Internet… and Minefield crashed.  Which was a great segue into “lets go look at these computers I brought and how I fix them!”

I brought three old Celeron “servers” (you can hardly call a Celeron a server) with lids off and we spent the next 20 minutes taking apart the machines.  We took out the computer’s brain and the fan to keep it cool (they didn’t believe me that you could cook food on the CPU when the computer was “thinking hard”).  We took out the two memory sticks and the hard drive and the IDE cable.  

These are things most parents in their right mind wouldn’t do with their home computer and these kids really enjoyed physically touching these parts and asking questions.

Mary Colvig helped me gather up a bunch of Firefox bags and my two kids and I had stuffed stickers into each one. Kids went crazy over the Foxkeh stickers and there was a collective “awwww!” when I showed them the “don’t hurt the web” stickers!

Anyways, good times. I enjoyed talking about Mozilla in a very different setting than I’m used to.

Who knows this logo?
Who knows this logo?
Minefield crashed with Kidzui
Minefield crashed with Kidzui
Inside a computer
Inside a computer
And this is a hard drive
And this is a hard drive

I’m going to go work on my new role in community outreach now…

Filed in Mozilla, Non Work | | Comments (10) »

How can IT reduce Firefox adoption/retention barriers? (Part II)

18-May-09

Nearly a month and a half ago I posted my first post on trying to figure out how IT can reduce adoption/retention barriers.

Over the past month we’ve worked out a different path than I originally thought we’d take.  Instead of targeting the Uninstall survey, I’ve been focusing my energies on point to explore #2:

2. How easily and quickly it is to get Firefox support using support.mozilla.com (is it too slow to be useful?)

I’ve been working closely with with David Tenser, Ken Kovash, and Seth Bindernagel (and all the l10n folks) to add a Kamplye survey to support.mozilla.com (SUMO). IT’s going to tag along with some survey questions David wanted to do. We hope to have something in place for June 1 (bug 489685).

I decided to target two major geographic regions (South America & Asia-Pacific) and have gotten help from our l10n community in translating the survey. You can follow our progress in bug 493096.

I’m including a sample mock-up below. I’m interested in any feedback you might have.

Kampyle / SUMO Survey Mock Up

Filed in Mozilla | | Comments Off

How can IT reduce Firefox adoption/retention barriers?

24-Mar-09

Why don’t you use Firefox and can IT change that?

At the beginning of the quarter I set an ambitious goal of trying to look at ways IT/Ops can reduce Firefox adoption/retention barriers.

I kept rotating on the best way to frame this question and in the process probably kept over thinking it. Mentally I tied this goal into looking at under-served geographic regions like South America (Brazil) and Asia-Pacific. I’m trying to answer these questions:

  1. How can IT help drive adoption/retention in Asia-Pacific or South America?
  2. What sort of local IT resources does it take to move the needle?
  3. If I put a data center in Brazil or Singapore, does it help?  Does it move the needle?  Do users care? (What’s my ROI?)

In other words, is there some function of Mozilla’s network or server infrastructure that prevents users in far away geographies from using Firefox?

I’ve been working with Asa Dotzler, Ken Kovash, Seth Bindernagel and Staś Małolepszy to try to figure out the right questions to ask and how to get them asked.

We came up with several points to explore:

  1. Web page load time for Mozilla properties (www.mozilla.com/www.mozilla.org) & other “participatory sites” (planet.mozilla.org, bugzilla.mozilla.org, developer.mozilla.org, labs.mozilla.com…)
  2. How easily and quickly it is to get Firefox support using support.mozilla.com (is it too slow to be useful?)
  3. Interactive browsing on addons.mozilla.org (is this site too slow to use?)
  4. Performance of AMO updates to extensions inside the Addons manager.

I had come up with three survey methods:

  1. Installer survey
  2. Uninstall survey
  3. Community surveys

The Installer survey is interesting but I’m not convinced that helps answer my questions.  At the point of that survey, the user hasn’t really used Firefox and any of those websites.  So I scratched it.

My goal for Phase 1 is to target the Uninstall survey and relate the responses to the users geography.

Looking for suggestions – followups in comments or email.

Filed in Mozilla | | Comments (4) »

Fx 3.0.7 release & this morning’s network performance issues

05-Mar-09

In computers systems (and with others) there are often bottlenecks and removing those often reveals new ones. Today’s an example of just that.

During a normal release we have tools we can use to adjust the rate at which we offer updates. We use this to reduce load on the back end systems or to help reduce load on the download mirrors.

Our preference is to do a release completely unthrottled so users get timely updates.

During the Firefox 3.0.6 release we had a number of system problems that prevented us from releasing updates unthrottled. These were all detailed in the Post Mortem.

To the Operations Team’s credit (and I’m serious here), most of those issues were removed prior to yesterday’s Firefox 3.0.7 release and by 9am this morning we were cranking along – no throttling.

Unfortunately the Mirror Network started showing pressure and instead of throttling back on the release, we opted to augment the Mirror Network with our own download servers in San Jose.

That pushed our aggregate bandwidth out of San Jose to nearly 3Gbps:

Global Bandwidth over 2Gbps

At around this time offsite monitors starting alerting about a sharp increase in page load times to various Mozilla website properties. Took a bit to track down but the newly turned up Level 3 peer was saturated:

Level3

Any outbound traffic whose best route was out through Level3 was impacted. We fixed this temporarily by turning down Level3.

(I should note that our design requirements for upstream transit is at least two connections per provider so we can push 2Gbps. Level 3 is no exception, however, the second connection has been offline because Derek was seeing a lot of packet loss across the optical connection which coincidentally got resolved today.)

These problems are solvable and we’ve had plans to put tools in place to balance load during situations like this. Unfortunately, today’s issues came up a lot quicker than we had planned.

A couple things we’ll be looking at before the next release:

  1. Evaluating Internap’s FCP to dynamically shift traffic based on cost and performance metrics. (And as luck would have it, this showed up this afternoon!)
  2. Looking to see how we can better balance outbound traffic outside of using FCP.
  3. Adding capacity to our Mirror Network (can you help?).
  4. Evaluating options around upgrading from several 1GE upstream connections to 10GE connections.

This is a great problem to have, to be sure, and a far cry from the panic three years ago of “OMG we’re about to push 100Mbps!”.

I’m really interested in how others have gone about solving problems like this. Leave me comments.

Filed in Mozilla, networking | Tagged | Comments (6) »

Math, when 2 equals .2

03-Mar-09

In my last post I talked about a 2mW user.  Specifically I said:

5. Or, with roughly 250 million Firefox users, 50kW is 2 milliwatts/user (or 144Wh/month).

I haven’t stopped thinking about that number as a metric of measuring efficiency or how much power it takes to do work (provide services) for some number of Firefox users.  I thought it’d be more interesting to trend wattage/user over time than just raw wattage (which is still too abstract for me to wrestle with).

My only problem is that my graphing tool kept spitting about numbers like:

241.24 μW

I had to stare at it bit before it clicked.

50kW/250m users = .0002 watts/user

50,000 W/250,000,000 users = .0002 watts/user

That comes out to .2mW/user (or 200μW/user). Quite frankly I’m embarrassed I didn’t catch my mistake earlier. I should have used something like this or this before posting!

_____________________

The milliwatt (symbol:mW) is equal to one thousandth (10-3) of a watt. The microwatt (symbol:μW) is equal to one millionth (10-6) of a watt.

Filed in Mozilla, Uncategorized | | Comments (4) »

The power of Mozilla = .2mW/user

25-Feb-09

We talk a lot about Data.  It’s a fundamental component of Mozilla’s 2010 Goals, second only to Mozilla’s role as a centerpiece of the Internet.  In September, Mitchell talked about “usage data.”

IT, and especially Operations, collects of a lot of Usage Data. Most of this Usage Data is trended and used in capacity planning and debugging.  It’s useful internally and probably not that interesting externally.

A lot of the more interesting data comes in the form of bandwidth graphs, NetFlow analysis, DNS statistics,  load balancer and router performance graphs, China bandwidth and China GSLB traffic distubution.

We’ve recently begun trending power usage from Mozilla’s primary data center in San Jose1.  The following graph shows wattage per row2:

San Jose power consumption

The genesis of this was inspired by Nobel Laureate Al Gore. I had the opportunity to see Mr. Gore give the keynote at the RSA Conference in San Francisco last April. His comment, “Make the invisible, visible,” resonated and stuck with me.

On average we’re using about 50kW (and remember, that’s a point-in-time measurement, the amount of power we’re using at any given time). I find that number to be too abstract — I’ll try to put that number into more meaningful terms (this calculator was helpful):

  1. If the local utility were to bill us for this, we’d get a bill for about 36MWh.  For perspective, my PG&E bill for last month was for 484kWh.
  2. The Tesla Roadster could go about 370km (230 miles) with 50kWh.
  3. In California, 1 kWh of electricity generates 0.878707 lbs of CO2 emissions.  Our San Jose data center generates  43.94 lbs of CO2 (or 0.02 metric tons).  For comparison, an average car (20 mpg) generates 44 lbs of CO2 after driving about 45 miles.3
  4. According to the DOE, the average U.S. home uses approximately 936kWh of electricity per month, which is about 31 kWh/day.  Daily we consume 1.2MWh.  At that rate we could power ~38 homes every day.
  5. Or, with roughly 250 million Firefox users, 50kW is 200 milliwattsmicrowatts/user (.2mW) (or 144Wh/month).

That last one is an interesting metric to measure power usage.  It’s more a measure of efficiency, efficiency per user.

I’m left with some questions:

  1. Who else is showing this sort of energy usage data?
  2. Is this a useful metric?  Can I use this to compare with others, apples-to-apples?  Does it mean anything?
  3. Is Mozilla’s data center energy usage good?  Bad?
  4. How can we use this data to help drive business decisions?
  5. Does energy consumption change during Firefox releases?

Lastly, I want to point out that the graph on this page is a link to a live image.  The data for this graph is pulled every five minutes and the graph image is updated twice an hour.  At some point this blog post will get lost in my archives so I’ve created a Wattage page with this graph and more historical graphs.  As we’re able to, I’ll add data from Mozilla’s other data centers.

_____________________

1 Admittedly, this graph under reports the true energy impact. It doesn’t take into account the amount of energy required to cool the data center space we’re in and is obviously missing both the Amsterdam and Beijing data centers.

2 This per-row graph shows something interesting. 16/101 has 9 racks and 212 devices in inventory. 16/102 has 6 racks, two of which are just patch panels. All the routing and load balancing hardware is in this row and inventory shows 59 devices. 16/103 has only 7 racks and 231, the most dense row and from the graph, less power hungry than 16/101. It’s also the row with all 141 Mac Minis.

3Calculations provided by Mira @ TerraPass.

Filed in Mozilla | Tagged , | Comments (9) »

More traffic analysis with NetFlow

19-Feb-09

Bandwidth usage by port

I’ve been working on a couple capacity planning projects and have been knee deep in bandwidth metrics. That, combined with turning up Level3 yesterday got me looking more into where that bandwidth is coming from (and Reed asked).

The first chart shows a breakdown by protocol. Shouldn’t be any surprise that 82% of Mozilla’s traffic is web related (SSL being the larger which, of course, make sense since it is out to destroy me). The “Other” category was filled with services under 1%.

Bandwidth usage by site
I took a look at the same bandwidth data and broke it down by source.  Most of the traffic is from addons.mozilla.org (which isn’t surprising – it alone causes all my SSL scaling headaches).

The other sites aren’t too surprising to me after a couple rounds of load balancer testing.  I expected and do see fxfeeds.mozilla.com (Firefox Live Bookmarks), versioncheck.addons.mozilla.org and services.addons.mozilla.org.

Filed in Mozilla, Uncategorized, networking | | Comments (1) »

Level3, post BGP turn up

18-Feb-09

Derek turned up BGP peering with Level3 this morning out of San Jose (this was supposed to be part of last night’s maintenance but he ran into turn up problems on Level3’s side).
sj-outboundbw

We’ve grown (bandwidth-wise) to the extent that having two transit providers wasn’t sufficient. Should either fail, I wouldn’t have felt comfortable pushing ~800Mbps out a single provider without any backup.

The trick will be making sure to hit bandwidth commits across all three…

Filed in Mozilla, networking | | Comments Off

Where is my Mozilla Firefox Infectious art?

03-Feb-09

My pleas to Mary have gone unheeded.  Where is my Mozilla Firefox themed Infectious artwork?

(Yes, this is a desperate plea to the community to come to my aid.)

Filed in Mozilla, Uncategorized | | Comments (2) »

10

29-Jan-09

Facebook says 25, planet.mozilla.org says 7. I prefer not to be a strict follower of either and I’m sure no one needs to re-read the rules again.  You can blame Mayu and Sean.

  1. If I had to do it all over again, I’d do something around sociology.  I’m often fascinated by human interactions and behavior.  I remember taking the bus in college and wearing earphones but having my Walkman (link for those too young to follow) off to watch & listen to others on the bus.
  2. History.  Hated this in high school, didn’t bother with it in college.  Now I’m fascinated with history, mostly Spanish California, early American history and some pre-history (like “Guns, Germs & Steel“). I tend to lose interest right around WWII.
  3. Both my kids, but my son in particular, love watching anything to do with science on TV with me.  This includes anything from Discovery Channel, Animal Planet or even History Channel.  And those are always good times.
  4. I did my first double century (200 mile bike ride) two years ago and haven’t done much riding since.  I have a goal of doing the Furnace Creek 508 before I’m 40 so I better get training.
  5. I was born in Denver and moved to Orange, CA until third grade when we moved to Hoffman Estates, IL (Chicagoland).  I moved out on my own and back to California (Bay Area) in January of 2006.  I’ve lived in Mountain View, Morgan Hill and Dublin, CA.  I relocated for work to Southern California in 1999 (also in January) and lived in Costa Mesa, Dana Point, Laguna Niguel and Aliso Viejo before moving back to the Bay Area in 2006 (this time not in January).  Southern California is definitely the best.
  6. Moving out of the house and 2200 miles away was one of the hardest thing I’ve done in my life.  It was weird to know that no matter what, things would never be the same.  My youngest brother was only five and I don’t know him nearly as much as I wish I did.
  7. My fondest memory of SoCal while growing up was the Angles Stadium.  It wasn’t that far from where we lived and I could always see the big A on the other side of the freeway.
  8. Met my wife in Irvine.  We worked at the same ISP – she was in sales and I was a Solaris admin learning to be a network engineer.  As hard as moving away from Illinois was, this alone made it worth it.
  9. I’m part Persian (by marriage).  This of course means I’m no stranger to large family gatherings or being the only one in the room who hasn’t a clue what the conversation is about (I don’t grok Persian).  Means I’ve also learned the proper way to make hot tea (and have modified that to make my own iced tea).
  10. Speaking of Persian, my two favorite dishes I don’t get often enough are Shirin Polo and Gheimeh.  But please keep anything with rose water away from me.
Filed in Non Work, Uncategorized | | Comments Off