That was two years ago.  Today’s steady state is around 600Mbps and it’s not uncommon to push closer to 1.5Gbps during a release.  We have a growing global presence with four data centers and have enough redundancy built in that it wasn’t any problem to lose one provider this past weekend (well, it was a problem but it wasn’t user impacting).

The environment has grown from a bunch of switches strung together with a mess of cables to an orderly mess of cables and a switching infrastructure that’s allowed us to be more nimble and do more complicated things more easily, often without ever having to physically visit the data center(s). And there are something like 51 app servers.

The network has grown and I need help.  I’m looking for someone to join the team and continue to grow and support Mozilla’s network and systems infrastructure (job description is after the jump).

If you’re ready for the challenge and opportunity to serve a community of 200 million Firefox users, send an email to careers at mozilla dot com!

Network Engineer Job Description

As a member of our IT team, you will assume a pivotal role in creating the company’s core high-volume systems and network infrastructure and participate in key design decisions. You will be expected to come up to speed quickly to meet technical goals and challenges and share a leadership role in a hard-working and collaborative team. We have high expectations and are looking for a seasoned professional with experience in a wide range of areas. Your time will be split between pure networking and assisting with Mozilla’s growing Linux & ESX infrastructure.

The network environment consists of Cisco and HP switches and routers, Citrix Netscaler load balancers, and Cisco and Juniper firewalls.

Requirements:

You must be self-motivated, capable of managing your time well, and work efficiently without close supervision. You place a high value on secure, highly available, fault-tolerant systems. You are proactive in identifying and resolving technical challenges, enthusiastically troubleshoot problems when they occur, and thrive as a collaborative team player. Key duties include:

• Provide support in the operation of Mozilla’s growing global network infrastructure.
• Support Mozilla’s corporate network and remote offices.
• Monitor system stability and performance.
• Ensure 24×7 operations.
• Act as an externally-facing point of contact to facilitate handling of problem reports, and maintain relations with network peers and vendors.
• Act as an internally-facing point of contact to escalate technical issues, and communicate network status.

Job Skill Requirements:

• Bachelor’s degree in a technical discipline (or equivalent work in IT related field).
• 3+ years of experience with enterprise/IT level network infrastructure and/or ISP network operations center/tier 1-2 support.
• In-depth knowledge of TCP/IP fundamentals (including Layers 2-7 content switching).
• Creative problem solving abilities.
• Network routing protocol (OSPF/BGP) knowledge.
• Network certifications such as CCNP/JNCIA/JNCIP (or equivalent training/experience) preferred but not required.
• Strong knowledge of datacenter design and layout.
• Ability to document and update processes.
• Experience with standard network change management and configuration policies.
• Experience with Unix/Linux administration is required.
• Experience and flexibility regarding on-call responsibilities.
• Understanding of web application tiers (app, database, caching)
• Experience with scalability issues in both the network and application layers

Additional Skills Strongly Desired:

• Strong experience with OS deployment and automation
• Scripting/tools ability is a plus (Perl, Python or PHP)
• Familiarity with Cisco 6500, FWSM, Citrix Netscalers & Load Balancers
• Familiarity with iSCSI SANs

To whomever it may concern,

I am interested in your domain names firefox.com,mozillafirefox.com,mozilla.com and was wondering if you would consider selling them to me.

I would be willing to go through Escrow.com (or any reputable escrow service of your choice) for the transaction, so that you know you are in good hands.
Also, I would not ask you to purchase an appraisal. This is simply a genuine request to purchase your domain names.

If you are interested in selling, please give me your asking price.

Thank you

1. 80 freakin’ power bricks, power cords
2. 80 ethernet cables and no horizontal (or vertical even!) cable management trays
3. 80 freakin’ power bricks
4. Power cycle requests for non-remotely-management “servers“

  

But with 80 freakin’ power cords I’m sure to end up with something like this (on the right) but worse.  Unless convinced otherwise I’ll be using ServerTech’s Remote Power Management PDUs but the only way to get the outlet density I need is with 5 16 outlet PDUs which will eat up 10u of rack space (on the backside).

We’re finding all sorts of uses for this now and have moved several websites over to this and have a couple more websites/services (like IRC) scheduled for this.

Purely out of interest I graphed the number of queries per hour through geodns from the two nameservers (one in San Jose, another in Amsterdam) and got:

I was concerned that two VMs wouldn’t work and I’d be looking at needing physical hardware but that hasn’t turned out to be the case yet (barely pushes more than 100Kbps with very little CPU load).  Although, since I’ve recently become a VMware DRS fan (like this guy) we’ve already cloned the San Jose nameserver and will be adding that as a third nameserver.  Since adding another nameserver/VM splits the load I also think this will scale nicely.

“that people would keep in their house, and it would measure energy consumption in the home, so how much electricy you’re using.”

Unfortunately it doens’t actually do that (sort of).  I think she meant the PG&E Energy Orb which does, sort of (an Ambient Orb can subscribe to a PG&E demand-response channels).  It gets wireless data from PG&E and glows in response to PG&E’s system-wide energy availability but it doesn’t tell me anything about my home usage.

I really want Ambient’s EnergyJoule but I don’t live in NYC.  Or maybe something like the DYI Kyoto but I don’t like in the UK and can’t tell if that matters really.

I can’t get Al Gore’s call to “make the invisible visible” out of my consciousness and really want something like this for my house.  Any pointers?

ps.  Mr. Gore, can you stop by Mozilla next time you’re in Mountain View or Cupertino?  We’re across the street from Google.  Thanks!

releases.mozilla.org is handled through round-robin DNS.  Regardless where you are in the world, you’ll get a “random” release mirror that may or may not be anywhere close to you and because of various physical reasons might induce a lot of latency and may or may not take a long time to download (admittedly, non-interactive transfers are less worried about latency but sitting in New York and downloading from China is going to take a long time).

As a network guy, I like to get my content as close to users as possible.  I know from my two visits to China that getting Add-ons updates from European mirror blow especially when I’m mere miles from our China release mirror!

After looking at a couple commercial solutions we decided to build our own geodns solution using the latest version of BIND and two patches to add Maxmind’s GeoIP functionality and to replace the backend zonefiles with MySQL.  That last patch lets us do all sorts of interesting thinks like nailing off bug 406267 (easily anyways, without having to munge text files).

Mark’s been working on all of the CLI management end to enable us to add or disable mirrors.  Hopefully I can get him to blog about the technical details but I’ll just add that prior to this I didn’t know anything about SQL views.

So when are we going deploy this?  Glad you asked.

On Tuesday, May 27 we’ll flip the switch and change releases.mozilla.org to a CNAME to releases.geo.mozilla.com, which, until Tuesday, will be in various states of testing but should work should be so inclined to try.

Currently, of the 15 release mirrors, 9 are in the US and 2 are in China.  The initial rollout will only directly benefit our Chinese users by directing them to the two Mozilla Online release mirrors in China.  Later we’ll try to do something on a geo-regional basis (but first, we’d want more than 3 mirrors in EU- anyone?)

This is just the tip of the iceberg.  Globally we use Netscaler-based GSLB to load balance and direct users to the network-wise closest site.  However, not all Mozilla properties are behind the Netscaler (or need to be) and we haven’t had a good way to serve users from different data centers unless they were.

Down the road we planning on adding some sort of weighting (not all mirrors or sites are equal), query statistics (where are lookups coming from?) and some sort of Nagios integration to automatically disable/enable a record.

And hopefully users will rejoice!

But it’s not without its gripes.  Hey VMware - why didn’t you integrate that into VirtualCenter (or even the VI client)?  Your CLI is horrible and confusing to use and if it weren’t for a third-party Window app, Storage VMotion would be really hard to use.  It’s also too bad DRS doesn’t somehow tie into this so storage could be moved around automagically for best performance.  Though maybe that’s better left to the storage boxes, like how EqualLogic does it.

And while I’m venting, where’s my OSX VI client?  Seriously, I have to fire up a CPU/resource sucking Windows VM on my MacBook or RDP to a Windows server to use VC?  Come on.

Regardless of your political feelings, seeing Al Gore on stage was electifying. Funny too!

Gore spoke about emerging green tech and specifically about CO₂ and the fact that it’s invisible. It’s invisible to see and smell and invisible to the markets and market control since there aren’t any economic incentives to control it.

He talked about asking the right questions to solve these problems and related a story of an electronics company that, while trying to find a replacement to CFCs used in cleaning circuit boards, asked the question “why did they need to be cleaned in the first place?” and came up with the “no-clean” process.

What resonated most to me was his call to use information technology to “make the invisible visible.”  That’s not something I had thought of.  Feels like I should put some more thought into that.

ps.  He came across as very country, very Tennessee.  Even wore boots.  Totally worth playing hookie for a couple hours this afternoon!

Update: CNET has an excellent article on Gore’s talk and does a better job covering it than I did.

View’s awesome but weather’s been cloudy/raining so no cool pictures.  Takes some getting used to too.

To check this I ran two weeks’ worth of web server logs through awstats with the geoip plugin.  From this, I’ve learned the following:

1. I was wrong.  In this post I said that “connectivity to mainland China can often be congested and can induce a lot of latency.”  Round-trip times between San Jose and China are only 30ms more than they are to Amsterdam (195ms vs. 166ms).  I don’t see any of the congestion that I saw from either of the two hotels I stayed or at that I see from the China office.
2. We’re serving a large portion of the Asia-Pacific region out of our Beijing, China colo. Far more than I think any of us had thought and we’re doing it remarkably well (even Gomez says so)!  I’ve even received emails from folks in that region wondering why www.mozilla.com is so much quicker/closer than any of Mozilla’s other web properties.

Looking at those log hits through awstats and pushing those results into Numbers gets me this:

(The 5% USA appears to be a combination of errors in the allocation database (IP is registered to a US entity but traceroute shows otherwise) and a few US-based ISPs that, for whatever reason, keep hitting China.  Other is a lot of smaller hits to Asia-Pacific countries and some IP addresses that didn’t automatically resolve to a country.)

Since the color seperation might be hard to read, the raw numbers are here.

I’m generally very happy with these results. In the next few weeks we should be adding addons.mozilla.org and spinning up our geo-dns setup to start pushing releases.mozilla.org out of China too.  Stay tuned.