Thought I’d share some before-and-after notes.

There’s a chunk of missing time when border2 was offline and I was rebuilding the config but it sure was worth it. The only CPU spike was shortly after all my BGP peers came back up and there was the necessary BGP Scanner run.

A couple more before-and-after snapshots:

FIB Usage (look more at the %Used for IPv4 routes):

border2#show platform hardware capacity | beg L3
L3 Forwarding Resources
             FIB TCAM usage:                     Total        Used       %Used
                  72 bits (IPv4, MPLS, EoM)     245760      244699        100%
                 144 bits (IP mcast, IPv6)        8192        1498         18%

                     detail:      Protocol                    Used       %Used
                                  IPv4                      244699        100%
                                  MPLS                           0          0%
                                  EoM                            0          0%
                                  IPv6                        1495         18%

border2#show platform hardware capacity | beg L3
L3 Forwarding Resources
             FIB TCAM usage:                     Total        Used       %Used
                  72 bits (IPv4, MPLS, EoM)     802816      267860         33%
                 144 bits (IP mcast, IPv6)      122880        1496          1%

                     detail:      Protocol                    Used       %Used
                                  IPv4                      267860         33%
                                  MPLS                           0          0%
                                  EoM                            0          0%
                                  IPv6                        1493          1%

Maximum Routes (changed this with mls cef maximum-routes ip 768 and mls cef maximum-routes mpls 1 - I remain hopeful IPv6 will take off):

border2#show mls cef max
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4                - 239k
 MPLS                - 1k (default)
 IPv6 + IP Multicast - 8k (default)

border2#show mls cef max
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4                - 768k
MPLS                - 1k
IPv6 + IP Multicast - 120k (default)

border1 gets upgraded Thursday night. Hopefully last night wasn’t disruptive for anyone.

Things have changed.  I’ll highlight just a couple of them:

• Active Firefox users grew from roughly 20 million users to over 70 million
• Mozilla’s outbound traffic has grown from ~150Mbps to well over 800Mbps (and over 1.5Gbps during release periods)
• BGP routers on the Internet have grown from something around 200k to more than 250k

That last bullet point brings us to today.

The two BGP speaking routers in San Jose both have Sup32 (the “CPU” of the router) and they have a limit to the maximum number of routes they can hold in their FIB TCAM (”route lookup table”).  Routes that can’t fit in the FIB TCAM end up being forwarded in software at the cost of CPU.  The more traffic we push, the high the CPU tends to run and lately it’s been running close to the point of uncomfortable.

I’m routinely getting alert emails:

border1.sj.mozilla.com five minute load average 62% exceeds 60%
border2.sj.mozilla.com five minute load average 83% exceeds 60%

And from trend graphs, it’s quite obvious.

I will be upgrading the Sup32s this week to Sup720-3BXLs.  I plan on doing one Tuesday and the other Thursday.  For the most part, this should be non-user impacting.  Most of the headache is going to be in the backend, moving router interfaces around, moving cacti graphs around and updating aggregtate graphs.

CA Academy of Sciences, Living Roof

This is far from my normal Mozilla related posts but worthwhile enough that I feel like sharing it.

I took my two children to the California Academy of Sciences this weekend.  I wasn’t sure what to expect but since it’s new (or newly opened) and I enjoy these types out outings with my children, this become our weekend activity.

It’s a combination aquarium (which saves me a trip to Monterey), natural history museum, planetarium, and rain forest.  The rain forest is enclosed in a 4-story glass-like globe with a winding staircase that, I think, lead to the roof top. The line was too long to bother going in with two children who clearly wanted to run around and explore instead.  There was also a great outdoor area on the west side of the building where the kids had fun running around while having snacks.

One of the places I’ve missed from Chicago was the Field Museum. The natural history part wasn’t anything as large as Chicago’s but was good enough to make me want to go back.

The absolute best, however, was when at bed time when my five-year old son told me that today was the best day ever.  So thanks, Academy.  You’ll be getting my membership application soon.

Things learned:

1. Buy tickets beforehand.  The lines were really long and the Planetarium was sold out by the time we got there.
2. Let the kids loose and let them tell you where to go and what to look at.  Turns out they know how to explore best.
3. Get a membership.
4. Museum has guest WiFi.

I’ll give a little background first.

At Mozilla’s main campus I’m using a Cisco 3845 ISR with two NM-WLC Wireless LAN controllers and have a total of 9 APs covering two buildings.

I broadcast two SSIDs - a guest one and a WPA/WPA2 Enterprise one.  Both wireless networks are bridged through the ISR onto the appropriate wired network through a BVI.

Problem #1

My first issue was mostly around client authentication.  Mozilla has a heavy percentage of Mac users and most had some sort of issue authenticating.  This problem became worse when the MacBook Airs came out and with some of the new gen MacBook Pros.  None of the Airs could authentication and a large number of the Pros started failing.  And not a single iPhone could authenticate.

Cisco’s default response was to:

1. Update my wireless drivers on OSX
2. Update the firmware on the WLC

#1 is impossible, #2 I did and no fix.  Finally after a month of pushing and two days of bringing in Aruba gear to prove to Cisco it wasn’t an OSX issue, Cisco found a solution.  The default EAP timeout was set to one second with a one second retry.  You had one second to type your password correctly and you had one chance to retry it.  Changing both of those to something more reasonable resolved most of the issues for Airs, Pros and iPhones.

(I don’t believe this was well documented - it’s not exposed through the webui WLC interface either and took TAC a long time to come up with this recommendation. Look for config advanced eap identity-request-timeout & config advanced eap identity-request-retries.)

Problem #2

The second problem is more involved and has been a problem since day one but hasn’t really been end-user affecting.  Most users will notice that wired users can not see wireless users’ iTunes libraries (and visa versa).

That’s just a symptom of the problem. Anything that relies on mDNS/Bonjour fails to work between wired and wireless users, including finding network-based Time Machine servers.

This manifested itself again when certain users couldn’t sync their Things content with their iPhone.  In troubleshooting, we (Justin) noticed that it used multicast to try to find devices to sync with.

I’ve narrowed down the problem to the following:

1. multicast traffic is not forwarded intra-WLC or inter-WLC
2. mulitcast traffic is not bridged out the BVI

From a wired host I ran:

tcpdump -n ip multicast and ether host 00:17:f2:09:d8:ea

and am unable to see any multicast data from my wireless host (it’s entirely possible that I don’t understand mDNS or how to use tcpdump well enough to troubleshoot this either).  As best as I can tell, the WLC is configured to process multicast:

(BS-WLC01) >show network sum

RF-Network Name............................. mozilla
Web Mode.................................... Disable
Secure Web Mode............................. Enable
Secure Web Mode Cipher-Option High.......... Disable
Secure Shell (ssh).......................... Enable
Telnet...................................... Disable
Ethernet Multicast Mode..................... Enable   Mode: Mcast  239.0.1.2
Ethernet Broadcast Mode..................... Enable

Cisco appears to have no clue on this either. The last response from TAC on this was:

I checked our query and found no response as of this time. I researched and found no similar devices in combination related to the matter. Be assured that I will make necessary follow-up and will provide you an update as soon as I receive a reply.

This worked without problems when I had that Aruba hardware for a couple days so I know this is not an OSX client issue - I wasable to stream from my iTunes library on my MacBook (wireless, on Aruba) to my wired desktop.

Cisco, why is this so hard to get working?!

That was two years ago.  Today’s steady state is around 600Mbps and it’s not uncommon to push closer to 1.5Gbps during a release.  We have a growing global presence with four data centers and have enough redundancy built in that it wasn’t any problem to lose one provider this past weekend (well, it was a problem but it wasn’t user impacting).

The environment has grown from a bunch of switches strung together with a mess of cables to an orderly mess of cables and a switching infrastructure that’s allowed us to be more nimble and do more complicated things more easily, often without ever having to physically visit the data center(s). And there are something like 51 app servers.

The network has grown and I need help.  I’m looking for someone to join the team and continue to grow and support Mozilla’s network and systems infrastructure (job description is after the jump).

If you’re ready for the challenge and opportunity to serve a community of 200 million Firefox users, send an email to careers at mozilla dot com!

Network Engineer Job Description

As a member of our IT team, you will assume a pivotal role in creating the company’s core high-volume systems and network infrastructure and participate in key design decisions. You will be expected to come up to speed quickly to meet technical goals and challenges and share a leadership role in a hard-working and collaborative team. We have high expectations and are looking for a seasoned professional with experience in a wide range of areas. Your time will be split between pure networking and assisting with Mozilla’s growing Linux & ESX infrastructure.

The network environment consists of Cisco and HP switches and routers, Citrix Netscaler load balancers, and Cisco and Juniper firewalls.

Requirements:

You must be self-motivated, capable of managing your time well, and work efficiently without close supervision. You place a high value on secure, highly available, fault-tolerant systems. You are proactive in identifying and resolving technical challenges, enthusiastically troubleshoot problems when they occur, and thrive as a collaborative team player. Key duties include:

• Provide support in the operation of Mozilla’s growing global network infrastructure.
• Support Mozilla’s corporate network and remote offices.
• Monitor system stability and performance.
• Ensure 24×7 operations.
• Act as an externally-facing point of contact to facilitate handling of problem reports, and maintain relations with network peers and vendors.
• Act as an internally-facing point of contact to escalate technical issues, and communicate network status.

Job Skill Requirements:

• Bachelor’s degree in a technical discipline (or equivalent work in IT related field).
• 3+ years of experience with enterprise/IT level network infrastructure and/or ISP network operations center/tier 1-2 support.
• In-depth knowledge of TCP/IP fundamentals (including Layers 2-7 content switching).
• Creative problem solving abilities.
• Network routing protocol (OSPF/BGP) knowledge.
• Network certifications such as CCNP/JNCIA/JNCIP (or equivalent training/experience) preferred but not required.
• Strong knowledge of datacenter design and layout.
• Ability to document and update processes.
• Experience with standard network change management and configuration policies.
• Experience with Unix/Linux administration is required.
• Experience and flexibility regarding on-call responsibilities.
• Understanding of web application tiers (app, database, caching)
• Experience with scalability issues in both the network and application layers

Additional Skills Strongly Desired:

• Strong experience with OS deployment and automation
• Scripting/tools ability is a plus (Perl, Python or PHP)
• Familiarity with Cisco 6500, FWSM, Citrix Netscalers & Load Balancers
• Familiarity with iSCSI SANs

To whomever it may concern,

I am interested in your domain names firefox.com,mozillafirefox.com,mozilla.com and was wondering if you would consider selling them to me.

I would be willing to go through Escrow.com (or any reputable escrow service of your choice) for the transaction, so that you know you are in good hands.
Also, I would not ask you to purchase an appraisal. This is simply a genuine request to purchase your domain names.

If you are interested in selling, please give me your asking price.

Thank you

1. 80 freakin’ power bricks, power cords
2. 80 ethernet cables and no horizontal (or vertical even!) cable management trays
3. 80 freakin’ power bricks
4. Power cycle requests for non-remotely-management “servers“

  

But with 80 freakin’ power cords I’m sure to end up with something like this (on the right) but worse.  Unless convinced otherwise I’ll be using ServerTech’s Remote Power Management PDUs but the only way to get the outlet density I need is with 5 16 outlet PDUs which will eat up 10u of rack space (on the backside).

We’re finding all sorts of uses for this now and have moved several websites over to this and have a couple more websites/services (like IRC) scheduled for this.

Purely out of interest I graphed the number of queries per hour through geodns from the two nameservers (one in San Jose, another in Amsterdam) and got:

I was concerned that two VMs wouldn’t work and I’d be looking at needing physical hardware but that hasn’t turned out to be the case yet (barely pushes more than 100Kbps with very little CPU load).  Although, since I’ve recently become a VMware DRS fan (like this guy) we’ve already cloned the San Jose nameserver and will be adding that as a third nameserver.  Since adding another nameserver/VM splits the load I also think this will scale nicely.

“that people would keep in their house, and it would measure energy consumption in the home, so how much electricy you’re using.”

Unfortunately it doens’t actually do that (sort of).  I think she meant the PG&E Energy Orb which does, sort of (an Ambient Orb can subscribe to a PG&E demand-response channels).  It gets wireless data from PG&E and glows in response to PG&E’s system-wide energy availability but it doesn’t tell me anything about my home usage.

I really want Ambient’s EnergyJoule but I don’t live in NYC.  Or maybe something like the DYI Kyoto but I don’t like in the UK and can’t tell if that matters really.

I can’t get Al Gore’s call to “make the invisible visible” out of my consciousness and really want something like this for my house.  Any pointers?

ps.  Mr. Gore, can you stop by Mozilla next time you’re in Mountain View or Cupertino?  We’re across the street from Google.  Thanks!

releases.mozilla.org is handled through round-robin DNS.  Regardless where you are in the world, you’ll get a “random” release mirror that may or may not be anywhere close to you and because of various physical reasons might induce a lot of latency and may or may not take a long time to download (admittedly, non-interactive transfers are less worried about latency but sitting in New York and downloading from China is going to take a long time).

As a network guy, I like to get my content as close to users as possible.  I know from my two visits to China that getting Add-ons updates from European mirror blow especially when I’m mere miles from our China release mirror!

After looking at a couple commercial solutions we decided to build our own geodns solution using the latest version of BIND and two patches to add Maxmind’s GeoIP functionality and to replace the backend zonefiles with MySQL.  That last patch lets us do all sorts of interesting thinks like nailing off bug 406267 (easily anyways, without having to munge text files).

Mark’s been working on all of the CLI management end to enable us to add or disable mirrors.  Hopefully I can get him to blog about the technical details but I’ll just add that prior to this I didn’t know anything about SQL views.

So when are we going deploy this?  Glad you asked.

On Tuesday, May 27 we’ll flip the switch and change releases.mozilla.org to a CNAME to releases.geo.mozilla.com, which, until Tuesday, will be in various states of testing but should work should be so inclined to try.

Currently, of the 15 release mirrors, 9 are in the US and 2 are in China.  The initial rollout will only directly benefit our Chinese users by directing them to the two Mozilla Online release mirrors in China.  Later we’ll try to do something on a geo-regional basis (but first, we’d want more than 3 mirrors in EU- anyone?)

This is just the tip of the iceberg.  Globally we use Netscaler-based GSLB to load balance and direct users to the network-wise closest site.  However, not all Mozilla properties are behind the Netscaler (or need to be) and we haven’t had a good way to serve users from different data centers unless they were.

Down the road we planning on adding some sort of weighting (not all mirrors or sites are equal), query statistics (where are lookups coming from?) and some sort of Nagios integration to automatically disable/enable a record.

And hopefully users will rejoice!