During a normal release we have tools we can use to adjust the rate at which we offer updates. We use this to reduce load on the back end systems or to help reduce load on the download mirrors.

Our preference is to do a release completely unthrottled so users get timely updates.

During the Firefox 3.0.6 release we had a number of system problems that prevented us from releasing updates unthrottled. These were all detailed in the Post Mortem.

To the Operations Team’s credit (and I’m serious here), most of those issues were removed prior to yesterday’s Firefox 3.0.7 release and by 9am this morning we were cranking along – no throttling.

Unfortunately the Mirror Network started showing pressure and instead of throttling back on the release, we opted to augment the Mirror Network with our own download servers in San Jose.

That pushed our aggregate bandwidth out of San Jose to nearly 3Gbps:

At around this time offsite monitors starting alerting about a sharp increase in page load times to various Mozilla website properties. Took a bit to track down but the newly turned up Level 3 peer was saturated:

Any outbound traffic whose best route was out through Level3 was impacted. We fixed this temporarily by turning down Level3.

(I should note that our design requirements for upstream transit is at least two connections per provider so we can push 2Gbps. Level 3 is no exception, however, the second connection has been offline because Derek was seeing a lot of packet loss across the optical connection which coincidentally got resolved today.)

These problems are solvable and we’ve had plans to put tools in place to balance load during situations like this. Unfortunately, today’s issues came up a lot quicker than we had planned.

A couple things we’ll be looking at before the next release:

1. Evaluating Internap’s FCP to dynamically shift traffic based on cost and performance metrics. (And as luck would have it, this showed up this afternoon!)
2. Looking to see how we can better balance outbound traffic outside of using FCP.
3. Adding capacity to our Mirror Network (can you help?).
4. Evaluating options around upgrading from several 1GE upstream connections to 10GE connections.

This is a great problem to have, to be sure, and a far cry from the panic three years ago of “OMG we’re about to push 100Mbps!”.

I’m really interested in how others have gone about solving problems like this. Leave me comments.

I’ve been working on a couple capacity planning projects and have been knee deep in bandwidth metrics. That, combined with turning up Level3 yesterday got me looking more into where that bandwidth is coming from (and Reed asked).

The first chart shows a breakdown by protocol. Shouldn’t be any surprise that 82% of Mozilla’s traffic is web related (SSL being the larger which, of course, make sense since it is out to destroy me). The “Other” category was filled with services under 1%.

I took a look at the same bandwidth data and broke it down by source.  Most of the traffic is from addons.mozilla.org (which isn’t surprising – it alone causes all my SSL scaling headaches).

The other sites aren’t too surprising to me after a couple rounds of load balancer testing.  I expected and do see fxfeeds.mozilla.com (Firefox Live Bookmarks), versioncheck.addons.mozilla.org and services.addons.mozilla.org.

We’ve grown (bandwidth-wise) to the extent that having two transit providers wasn’t sufficient. Should either fail, I wouldn’t have felt comfortable pushing ~800Mbps out a single provider without any backup.

The trick will be making sure to hit bandwidth commits across all three…

We ran into several issues and postponed the upgrade on core2 until those issues are resolved.

1. The switch’s Compact Flash cards weren’t formatted in a format that rommon (the low level boot loader) could read and the switch failed to load any OS when rebooted (bug 473084).
   

   <rant>
   Unfortunately IOS didn’t flag that as an error when reading/writing to it. It didn’t even flag an error when the boot variable was set to boot off of it which is stupid because IOS clearly knew as shown in the log when I had remote-hands re-seat the card:

   Jan 11 22:44:23 core2 3927937: Jan 11 22:44:21.978 PDT: %PCMCIAFS-SP-5-DIBERR: PCMCIA disk 0 is formatted from a different router or PC. A format in this router is required before an image can be booted from this device
   </rant>

2. Two of the VMware ESX storage arrays are single-homed, connected to one switch (bug 473113).  This is fallout from previous NetApp performance issues that were forgotten and never addressed and caused a number of build VMs to go offline (bug 473112) .
3. A number of non-user facing, multi-homed hosts went offline.  All of the RHEL Linux servers have an active/standby network setup.  In several cases the standby interface didn’t work or wasn’t properly configured.  This was more of an annoyance to the IT Team than to anyone else but did cause outages for some backend services (most notably the VMware VC server and mradm01, one of the Nagios servers).

We’ll be addressing those issues before scheduling the remaining upgrade to core2. We’ll also be looking at implementing some routine (perhaps quarterly) test of the infrastructure in a controlled environment to ensure its high “availbility-ness”.

Tonight’s one of those rare exceptions. Derek is upgrading both core1 and core2 to pickup software support for Cisco’s ACE module. We’re planning on using this in conjunction with the Zeus ZXTM cluster we setup.

A little sad though.  It’s going to reset uptime:

core1#sh ver | inc uptime
core1 uptime is 2 years, 28 weeks, 2 days, 20 hours, 40 minutes

core2#sh ver | inc uptime
core2 uptime is 2 years, 28 weeks, 2 days, 20 hours, 51 minutes

Love ‘em or hate ‘em, Cisco gear is solid.

Thought I’d share some before-and-after notes.

There’s a chunk of missing time when border2 was offline and I was rebuilding the config but it sure was worth it. The only CPU spike was shortly after all my BGP peers came back up and there was the necessary BGP Scanner run.

A couple more before-and-after snapshots:

FIB Usage (look more at the %Used for IPv4 routes):

border2#show platform hardware capacity | beg L3
L3 Forwarding Resources
             FIB TCAM usage:                     Total        Used       %Used
                  72 bits (IPv4, MPLS, EoM)     245760      244699        100%
                 144 bits (IP mcast, IPv6)        8192        1498         18%

                     detail:      Protocol                    Used       %Used
                                  IPv4                      244699        100%
                                  MPLS                           0          0%
                                  EoM                            0          0%
                                  IPv6                        1495         18%

border2#show platform hardware capacity | beg L3
L3 Forwarding Resources
             FIB TCAM usage:                     Total        Used       %Used
                  72 bits (IPv4, MPLS, EoM)     802816      267860         33%
                 144 bits (IP mcast, IPv6)      122880        1496          1%

                     detail:      Protocol                    Used       %Used
                                  IPv4                      267860         33%
                                  MPLS                           0          0%
                                  EoM                            0          0%
                                  IPv6                        1493          1%

Maximum Routes (changed this with mls cef maximum-routes ip 768 and mls cef maximum-routes mpls 1 – I remain hopeful IPv6 will take off):

border2#show mls cef max
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4                - 239k
 MPLS                - 1k (default)
 IPv6 + IP Multicast - 8k (default)

border2#show mls cef max
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4                - 768k
MPLS                - 1k
IPv6 + IP Multicast - 120k (default)

border1 gets upgraded Thursday night. Hopefully last night wasn’t disruptive for anyone.

Things have changed.  I’ll highlight just a couple of them:

• Active Firefox users grew from roughly 20 million users to over 70 million
• Mozilla’s outbound traffic has grown from ~150Mbps to well over 800Mbps (and over 1.5Gbps during release periods)
• BGP routers on the Internet have grown from something around 200k to more than 250k

That last bullet point brings us to today.

The two BGP speaking routers in San Jose both have Sup32 (the “CPU” of the router) and they have a limit to the maximum number of routes they can hold in their FIB TCAM (“route lookup table”).  Routes that can’t fit in the FIB TCAM end up being forwarded in software at the cost of CPU.  The more traffic we push, the high the CPU tends to run and lately it’s been running close to the point of uncomfortable.

I’m routinely getting alert emails:

border1.sj.mozilla.com five minute load average 62% exceeds 60%
border2.sj.mozilla.com five minute load average 83% exceeds 60%

And from trend graphs, it’s quite obvious.

I will be upgrading the Sup32s this week to Sup720-3BXLs.  I plan on doing one Tuesday and the other Thursday.  For the most part, this should be non-user impacting.  Most of the headache is going to be in the backend, moving router interfaces around, moving cacti graphs around and updating aggregtate graphs.

I’ll give a little background first.

At Mozilla’s main campus I’m using a Cisco 3845 ISR with two NM-WLC Wireless LAN controllers and have a total of 9 APs covering two buildings.

I broadcast two SSIDs – a guest one and a WPA/WPA2 Enterprise one.  Both wireless networks are bridged through the ISR onto the appropriate wired network through a BVI.

Problem #1

My first issue was mostly around client authentication.  Mozilla has a heavy percentage of Mac users and most had some sort of issue authenticating.  This problem became worse when the MacBook Airs came out and with some of the new gen MacBook Pros.  None of the Airs could authentication and a large number of the Pros started failing.  And not a single iPhone could authenticate.

Cisco’s default response was to:

1. Update my wireless drivers on OSX
2. Update the firmware on the WLC

#1 is impossible, #2 I did and no fix.  Finally after a month of pushing and two days of bringing in Aruba gear to prove to Cisco it wasn’t an OSX issue, Cisco found a solution.  The default EAP timeout was set to one second with a one second retry.  You had one second to type your password correctly and you had one chance to retry it.  Changing both of those to something more reasonable resolved most of the issues for Airs, Pros and iPhones.

(I don’t believe this was well documented – it’s not exposed through the webui WLC interface either and took TAC a long time to come up with this recommendation. Look for config advanced eap identity-request-timeout & config advanced eap identity-request-retries.)

Problem #2

The second problem is more involved and has been a problem since day one but hasn’t really been end-user affecting.  Most users will notice that wired users can not see wireless users’ iTunes libraries (and visa versa).

That’s just a symptom of the problem. Anything that relies on mDNS/Bonjour fails to work between wired and wireless users, including finding network-based Time Machine servers.

This manifested itself again when certain users couldn’t sync their Things content with their iPhone.  In troubleshooting, we (Justin) noticed that it used multicast to try to find devices to sync with.

I’ve narrowed down the problem to the following:

1. multicast traffic is not forwarded intra-WLC or inter-WLC
2. mulitcast traffic is not bridged out the BVI

From a wired host I ran:

tcpdump -n ip multicast and ether host 00:17:f2:09:d8:ea

and am unable to see any multicast data from my wireless host (it’s entirely possible that I don’t understand mDNS or how to use tcpdump well enough to troubleshoot this either).  As best as I can tell, the WLC is configured to process multicast:

(BS-WLC01) >show network sum

RF-Network Name............................. mozilla
Web Mode.................................... Disable
Secure Web Mode............................. Enable
Secure Web Mode Cipher-Option High.......... Disable
Secure Shell (ssh).......................... Enable
Telnet...................................... Disable
Ethernet Multicast Mode..................... Enable   Mode: Mcast  239.0.1.2
Ethernet Broadcast Mode..................... Enable

Cisco appears to have no clue on this either. The last response from TAC on this was:

I checked our query and found no response as of this time. I researched and found no similar devices in combination related to the matter. Be assured that I will make necessary follow-up and will provide you an update as soon as I receive a reply.

This worked without problems when I had that Aruba hardware for a couple days so I know this is not an OSX client issue – I wasable to stream from my iTunes library on my MacBook (wireless, on Aruba) to my wired desktop.

Cisco, why is this so hard to get working?!

That was two years ago.  Today’s steady state is around 600Mbps and it’s not uncommon to push closer to 1.5Gbps during a release.  We have a growing global presence with four data centers and have enough redundancy built in that it wasn’t any problem to lose one provider this past weekend (well, it was a problem but it wasn’t user impacting).

The environment has grown from a bunch of switches strung together with a mess of cables to an orderly mess of cables and a switching infrastructure that’s allowed us to be more nimble and do more complicated things more easily, often without ever having to physically visit the data center(s). And there are something like 51 app servers.

The network has grown and I need help.  I’m looking for someone to join the team and continue to grow and support Mozilla’s network and systems infrastructure (job description is after the jump).

If you’re ready for the challenge and opportunity to serve a community of 200 million Firefox users, send an email to careers at mozilla dot com!

Network Engineer Job Description

As a member of our IT team, you will assume a pivotal role in creating the company’s core high-volume systems and network infrastructure and participate in key design decisions. You will be expected to come up to speed quickly to meet technical goals and challenges and share a leadership role in a hard-working and collaborative team. We have high expectations and are looking for a seasoned professional with experience in a wide range of areas. Your time will be split between pure networking and assisting with Mozilla’s growing Linux & ESX infrastructure.

The network environment consists of Cisco and HP switches and routers, Citrix Netscaler load balancers, and Cisco and Juniper firewalls.

Requirements:

You must be self-motivated, capable of managing your time well, and work efficiently without close supervision. You place a high value on secure, highly available, fault-tolerant systems. You are proactive in identifying and resolving technical challenges, enthusiastically troubleshoot problems when they occur, and thrive as a collaborative team player. Key duties include:

• Provide support in the operation of Mozilla’s growing global network infrastructure.
• Support Mozilla’s corporate network and remote offices.
• Monitor system stability and performance.
• Ensure 24×7 operations.
• Act as an externally-facing point of contact to facilitate handling of problem reports, and maintain relations with network peers and vendors.
• Act as an internally-facing point of contact to escalate technical issues, and communicate network status.

Job Skill Requirements:

• Bachelor’s degree in a technical discipline (or equivalent work in IT related field).
• 3+ years of experience with enterprise/IT level network infrastructure and/or ISP network operations center/tier 1-2 support.
• In-depth knowledge of TCP/IP fundamentals (including Layers 2-7 content switching).
• Creative problem solving abilities.
• Network routing protocol (OSPF/BGP) knowledge.
• Network certifications such as CCNP/JNCIA/JNCIP (or equivalent training/experience) preferred but not required.
• Strong knowledge of datacenter design and layout.
• Ability to document and update processes.
• Experience with standard network change management and configuration policies.
• Experience with Unix/Linux administration is required.
• Experience and flexibility regarding on-call responsibilities.
• Understanding of web application tiers (app, database, caching)
• Experience with scalability issues in both the network and application layers

Additional Skills Strongly Desired:

• Strong experience with OS deployment and automation
• Scripting/tools ability is a plus (Perl, Python or PHP)
• Familiarity with Cisco 6500, FWSM, Citrix Netscalers & Load Balancers
• Familiarity with iSCSI SANs