5. Or, with roughly 250 million Firefox users, 50kW is 2 milliwatts/user (or 144Wh/month).

I haven’t stopped thinking about that number as a metric of measuring efficiency or how much power it takes to do work (provide services) for some number of Firefox users.  I thought it’d be more interesting to trend wattage/user over time than just raw wattage (which is still too abstract for me to wrestle with).

My only problem is that my graphing tool kept spitting about numbers like:

241.24 μW

I had to stare at it bit before it clicked.

50kW/250m users = .0002 watts/user

50,000 W/250,000,000 users = .0002 watts/user

That comes out to .2mW/user (or 200μW/user). Quite frankly I’m embarrassed I didn’t catch my mistake earlier. I should have used something like this or this before posting!

_____________________

The milliwatt (symbol:mW) is equal to one thousandth (10^-3) of a watt. The microwatt (symbol:μW) is equal to one millionth (10^-6) of a watt.

I’ve been working on a couple capacity planning projects and have been knee deep in bandwidth metrics. That, combined with turning up Level3 yesterday got me looking more into where that bandwidth is coming from (and Reed asked).

The first chart shows a breakdown by protocol. Shouldn’t be any surprise that 82% of Mozilla’s traffic is web related (SSL being the larger which, of course, make sense since it is out to destroy me). The “Other” category was filled with services under 1%.

I took a look at the same bandwidth data and broke it down by source.  Most of the traffic is from addons.mozilla.org (which isn’t surprising – it alone causes all my SSL scaling headaches).

The other sites aren’t too surprising to me after a couple rounds of load balancer testing.  I expected and do see fxfeeds.mozilla.com (Firefox Live Bookmarks), versioncheck.addons.mozilla.org and services.addons.mozilla.org.

(Yes, this is a desperate plea to the community to come to my aid.)

1. If I had to do it all over again, I’d do something around sociology.  I’m often fascinated by human interactions and behavior.  I remember taking the bus in college and wearing earphones but having my Walkman (link for those too young to follow) off to watch & listen to others on the bus.
2. History.  Hated this in high school, didn’t bother with it in college.  Now I’m fascinated with history, mostly Spanish California, early American history and some pre-history (like “Guns, Germs & Steel“). I tend to lose interest right around WWII.
3. Both my kids, but my son in particular, love watching anything to do with science on TV with me.  This includes anything from Discovery Channel, Animal Planet or even History Channel.  And those are always good times.
4. I did my first double century (200 mile bike ride) two years ago and haven’t done much riding since.  I have a goal of doing the Furnace Creek 508 before I’m 40 so I better get training.
5. I was born in Denver and moved to Orange, CA until third grade when we moved to Hoffman Estates, IL (Chicagoland).  I moved out on my own and back to California (Bay Area) in January of 2006.  I’ve lived in Mountain View, Morgan Hill and Dublin, CA.  I relocated for work to Southern California in 1999 (also in January) and lived in Costa Mesa, Dana Point, Laguna Niguel and Aliso Viejo before moving back to the Bay Area in 2006 (this time not in January).  Southern California is definitely the best.
6. Moving out of the house and 2200 miles away was one of the hardest thing I’ve done in my life.  It was weird to know that no matter what, things would never be the same.  My youngest brother was only five and I don’t know him nearly as much as I wish I did.
7. My fondest memory of SoCal while growing up was the Angles Stadium.  It wasn’t that far from where we lived and I could always see the big A on the other side of the freeway.
8. Met my wife in Irvine.  We worked at the same ISP – she was in sales and I was a Solaris admin learning to be a network engineer.  As hard as moving away from Illinois was, this alone made it worth it.
9. I’m part Persian (by marriage).  This of course means I’m no stranger to large family gatherings or being the only one in the room who hasn’t a clue what the conversation is about (I don’t grok Persian).  Means I’ve also learned the proper way to make hot tea (and have modified that to make my own iced tea).
10. Speaking of Persian, my two favorite dishes I don’t get often enough are Shirin Polo and Gheimeh.  But please keep anything with rose water away from me.

You could argue I could have waited a couple more days but shaving at this scale is best done at home and not in a hotel!  Too bad too because it was just starting to get comfortable.

Here’s to next year!

Flash forward 30 some years and I find I’m doing the same with my two children and weirdly enough on Apple computers (first computer I owned was  Commdore 64 followed by an Amiga and then some Gateway running Windows in 1995 – only recently went back to Apple).  My son spends his time in KidZui (which to him is the same as pbskids.org) and my daugther does drawing with Tux Paint (on an 8 year-old Powerbook, but she doesn’t seem to notice or care).

My son recently asked me if he could have my black MacBook when he is older.  I laughed, sure, but I haven’t any idea what computers will look like 30 years from now!

For a number of months we’ve been looking at Zeus and their ZXTM product.  It has some advantages (and some disadvantages) and one of the biggest is that it’s all software and I can quickly deploy it on any available hardware we have.  Zeus has been extremely helpful in quickly issuing unlimited node eval license keys on short notice (so thanks Jasper/Chris) and providing the level of supported I’d more likely expect to be given to a paying customer.

We’ve shifted two of the highest traffic back-end infrastructure services for Firefox over to a Zeus ZXTM cluster:

1. fxfeeds.mozilla.org – Live Bookmarks feed URL in a default Firefox install.
2. versioncheck.addons.mozilla.org – URL Firefox uses to check for Add-On updates.

fxfeeds.mozilla.org is nothing more than an RSS feed.  The site itself has no real content – there isn’t even a DocumentRoot defined in Apache.  In fact, the config is nothing but 40 HTTP redirects:

[root@mradm02 domains]# egrep 'Redirect|Rewrite' fxfeeds.mozilla.org.conf  | wc -l
40

versioncheck.addons.mozilla.org is my one Firefox 3 claim-to-fame.  Without that change, a lot of the scalability work we’re doing would be extremely difficult and require a lot more QA time.

Since Firefox checks for Add-On updates after updating itself (and periodically on its own), versioncheck.addons.mozilla.org is one of the sites that sees a large spike in traffic around release time.  It’s also highly cachable content and easy to scale with CPU (for SSL) and memory (for cache).

I moved fxfeeds.mozilla.org over yesterday and was astonished to see what sort of traffic that site alone generated – staggering might be a better way to say it.  Nearly 40Mbps of basically 302 redirects and upwards of 400,000 connections/second (and incidentally, moving that site off the Netscalers significantly dropped its resources issues).

versioncheck.addons.mozilla.org was moved Monday morning onto a test ZXTM cluster and again later this afternoon on a more production capable ZXTM cluster.  It’ll push 30Mbps on its own during non-release periods and 2-3x that during a full release.

Since no post is good without pictures, I grabbed two graphs from the ZXTM cluster.  The first shows connections/second and the other shows bandwidth (bits per second).

Things have changed.  I’ll highlight just a couple of them:

• Active Firefox users grew from roughly 20 million users to over 70 million
• Mozilla’s outbound traffic has grown from ~150Mbps to well over 800Mbps (and over 1.5Gbps during release periods)
• BGP routers on the Internet have grown from something around 200k to more than 250k

That last bullet point brings us to today.

The two BGP speaking routers in San Jose both have Sup32 (the “CPU” of the router) and they have a limit to the maximum number of routes they can hold in their FIB TCAM (”route lookup table”).  Routes that can’t fit in the FIB TCAM end up being forwarded in software at the cost of CPU.  The more traffic we push, the high the CPU tends to run and lately it’s been running close to the point of uncomfortable.

I’m routinely getting alert emails:

border1.sj.mozilla.com five minute load average 62% exceeds 60%
border2.sj.mozilla.com five minute load average 83% exceeds 60%

And from trend graphs, it’s quite obvious.

I will be upgrading the Sup32s this week to Sup720-3BXLs.  I plan on doing one Tuesday and the other Thursday.  For the most part, this should be non-user impacting.  Most of the headache is going to be in the backend, moving router interfaces around, moving cacti graphs around and updating aggregtate graphs.

CA Academy of Sciences, Living Roof

This is far from my normal Mozilla related posts but worthwhile enough that I feel like sharing it.

I took my two children to the California Academy of Sciences this weekend.  I wasn’t sure what to expect but since it’s new (or newly opened) and I enjoy these types out outings with my children, this become our weekend activity.

It’s a combination aquarium (which saves me a trip to Monterey), natural history museum, planetarium, and rain forest.  The rain forest is enclosed in a 4-story glass-like globe with a winding staircase that, I think, lead to the roof top. The line was too long to bother going in with two children who clearly wanted to run around and explore instead.  There was also a great outdoor area on the west side of the building where the kids had fun running around while having snacks.

One of the places I’ve missed from Chicago was the Field Museum. The natural history part wasn’t anything as large as Chicago’s but was good enough to make me want to go back.

The absolute best, however, was when at bed time when my five-year old son told me that today was the best day ever.  So thanks, Academy.  You’ll be getting my membership application soon.

Things learned:

1. Buy tickets beforehand.  The lines were really long and the Planetarium was sold out by the time we got there.
2. Let the kids loose and let them tell you where to go and what to look at.  Turns out they know how to explore best.
3. Get a membership.
4. Museum has guest WiFi.