–

As many of you already know – we had some pretty serious issues over the past weeks with the storage system that supports the build/unit test environment. We have resolved the issues and wanted to give everyone a run down of the issues that we found, what we have done to resolve them and what open tasks are left.

The issue manifested itself in a few ways. We saw slow transfers, scsi aborts, reservation failures and VM guest level corruption. This started as a very rare occurrence and over time became more and more frequent to the point that we could not keep a small number of i/o intensive VMs up for 1 hour and had trouble getting them off. We started troubleshooting the issue a few weeks ago, and finally came to a total resolution early this week. Here is a summary of the issues, how they came to be and how we resolved them:

* http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=226424 (A filer may exhibit poor performance due to WAFL holding on to too many network
buffers and not releasing them in a timely fashion.)
To fix this, we had to do an upgrade to 7.2.4 – that has been completed.

* NetApp LUN’s were created of the wrong LUN type.
This was caused by a error in the LUN creation workflow causing the LUN to be set to the default value (Solaris). 3 out of 4 LUNs were of type Solaris causing blocks to not be written efficiently to the disk (the 4k VMWare blocks were written offset to the true disk geometry). Reading of the LUN would cause many read aheads and at times overwhelm the filer due to the inefficient layout on disk. To remedy this we migrated data off, re-created all of the LUNs and re-migrated the data back.

* NetApp igroup’s set to the wrong type.
Initially Netapp advised that linux igroup type (what maps the LUN to various hosts) were OK for use with VMWare. This was incorrect causing improper scsi reservations and iscsi timeouts. NetApp is updating their internal documentation to reflect this change.

* Network setup issues
Initial setup from NetApp advised us to setup the network in a specific configuration (one link to each upstream switch with a virtual interface bonding them). After further investigation, I found this is *not* the best practice and in fact causing issues with dead HBA paths. To correct this temporarily, we disabled one of the links, having single uplinks (still with redundant heads)

All of these issues created major performance degradation and block level access/corruption problems. They have all been resolved at this point. We still need to adjust the network interfaces to be more redundant.

Special thanks to the release engineering team has been *incredibly* patient with us as we worked through this. I know how frustrating it was and they kept a smile (well, kind of) through the situation – really helped us keep pushing forward to a solution. Thanks also to mrz for the amazing amount of work he put into this…very dedicated to finding a solution no matter what time it was.

To mitigate this issue going forward, we have make two changes.

• Modified the port-channels connecting the core switches to downstream switches to better handle a port-channel member failure.
• We also further tuned broadcast storm protection on every switch port to limit the amount of broadcast & multicast traffic any one device is allowed to send.

Furthermore, we have a priority case open with the vendor to determine the cause of the issue as we did capture debug logs. This was in no way related to the scheduled downtime we were in, it just happened to coincide. We apologize for any inconvenience this may have caused. We’ll continue to follow up with the vendor to make sure this issue does not happen again.

We’d like to build up our mirror network to be even stronger! I am making a call to the community to help us find other mirror sources. Already Paul Vixie from the Internet Software Consortium has stepped up and donated 3gb/s of mirror peak capacity (!). Details on what is required can be found here: http://www.mozilla.org/mirroring.html. While we are always happy to take any mirror donation, we are specifically looking for mirrors which can handle in excess of 100mb/s during peak traffic times. Please contact me directly if you have any ideas of people/organizations/companies that might be willing to donate either bandwidth or mirror space.

We here at Mozilla are big fans of OpenVPN. When we rebuilt our datacenter, we did a large search for the right VPN solution. Mozilla’s requirements were somewhat specific:

* Had to work with all three platforms (mac, linux, windows)
* Needed to work with our LDAP infrastructure (i.e. not AD)
* Needed to work through NAT
* We needed to be able to give each user granular per-host access
* We wanted a solution that would allow just Mozilla traffic to traverse the VPN rather than forcing all traffic through the VPN

We looked at many options, most of which were commercial closed-source solutions (given the lack of options). Ideally, a client-less, SSL-based solution would have been ideal, but it was clear Firefox (!) and Mac support was not ready. We decided on OpenVPN as it met all of our requirements and had the added benifit of being open source and free!

We’ve been happily using openvpn with TunnelBlick as our mac client. Justdave even created a custom installer for our users (pretty slick Dave ). But along comes Leopard – with changes such that the low level network drivers don’t function anymore (along with other issues in the GUI). With some research, mrz found that a OS X tuntap development team just released new drivers which support Leopard. Still, openvpn won’t connect, TunnelBlick won’t run, etc, so this weekend I set out to fix the issues. After 3-4 hours of figuring out how the TunnelBlick build setup works, fixing some bugs and adding in the new drivers, I have a working version of TunnelBlick, openvpn and tuntap drivers on Leopard.

What’s the point of this rant? I could have *never* fixed this with a closed source VPN client. I’d be hamstrung by Cisco (yes, Cisco John) or some other network vendor while they gave me the normal story that Mac is not a large enough platform to dedicate resources too (nevermind that 90+% of Mozilla engineers use Mac hardware). Being able to look at the source, build system and composition of each of these apps made it possible to figure out what the issue was, fix it, and post this build for anyone else who needs it.

Makes me remember why what we do here at Mozilla is so important. So, if you need a Leopard version of TunnelBlick (with tuntap drivers and openvpn 2.0.9 with lzo support), here you go.

First a little background – for one 24 hour period, we would need to serve *all* en-US and de downloads which originate from our website – not a small number. We estimate ~500k downloads a day overall, with a large percentage being en-US and de. Why would we want to host the downloads when we have an excellent mirror network setup, happily serving up our bits? We were interested in gathering statistics on how many people started, aborted or completed the downloads. We could do some of this by adding an FTP server of our own into bouncer, but is much more interesting to get an idea of the behavior seeing *all* the traffic. Also, we can correlate the logs later to number of active users and website behavior. Plus 24 hours won’t kill my 95th percentile bandwidth bills

Second, seeing all of the traffic allows us to get a great view of the diversity, amount and frequency of downloads. As you’ll see below, it was quite an increase in our normal traffic.

Third, it’s a great test to stress test our infrastructure, verifying we don’t have any unexpected bottlenecks or performance issues. The good news here is the systems passed with flying colors.

Our setup was pretty simple – we built out three download servers with the archive.mozilla.org nfs share mounted. Slapped apache on them, added them to bouncer and we were off to the races. Here are the traffic graphs (you can probably tell when we switched things over):

Furthermore, Apache really impressed me. The servers were pushing upwards of 80mbs each off nfs, with a load of… 0.00 and cpu hovering around 5%. We sometimes got the occasional 0.10 spike, but all in all, pretty amazing. Graphs from one of the machines:

All in all, I was very happy with lack if impact on the systems and continued good performance.

We released at 3pm PDT yesterday – here are some of the stats so far:

* Just under 1 million people have been updated from 1.5.0.12 -> 2.0.0.4
* We are updating people at a rate of 20-30 per second (FF2 download rate was about 30/second)
* Mirrors are seeing about a 2-3x increase in bandwidth

I expect to see increased load throughout the day, but so far so good! Huge thanks to webdev with their help to optimize addons.mozilla.org – it’s been a huge win for this release.

Enter VMWare 3.0.

You might ask “why would you want to use the first rev a such a new release”? Good point – problem is, VMWare only supports one, that’s right one 4 port ethernet card (Intel quad port). No problem – I order our servers with the supported Intel PRO/1000 Quad Port Server Adapter. This adapter has worked fine in other older boxes so I have no reason to think it won’t work. Well, Intel has discontinued the “MT” version of the cards, and only sells the “GT” version of the cards. After 5 days of tedious interaction with VMware web support, we find why 2.5.3 won’t see the card – 2.5.3 only supports the out of date and impossible to get “MT” version of the card.

VMWare 2.5.3 only supports one quad port card that you can’t get, with no ETA on “GT” version support for 2.5.x – great – guess we’ll try 3.0. So we get 3.0 install and it recognizes the “GT” version of the card – yay.

Now the real fun starts – I go to the web admin console that I have used in the past to configure the host – wait – there is nothing here?!? No network configuration, no VM config options, nothing! It appears VMWare has moved all the configuration to a .Net, windows-only application called “Infrastructure Client”. I use a Mac, much of my team runs Linux. Why in the world would you take a web-based, platform independent configuration tool and move it to a .Net application?

So you ask why I need Parallels to run VMWare? That’s why (I’m on a Mac, and need a Windows OS to configure VMWare). And it sucks. I think VMWare has a really great product and one that can help scale our infrastructure, but why focus the admin tools on just one OS? Seems the whole company is built around virtualization and cross-platform support, but has taken a step backwards with this change in 3.0. Hopefully VMWare will fix this issue by producing upcoming versions of Infrastructure Client by supporting it on different platforms.