–

As many of you already know – we had some pretty serious issues over the past weeks with the storage system that supports the build/unit test environment. We have resolved the issues and wanted to give everyone a run down of the issues that we found, what we have done to resolve them and what open tasks are left.

The issue manifested itself in a few ways. We saw slow transfers, scsi aborts, reservation failures and VM guest level corruption. This started as a very rare occurrence and over time became more and more frequent to the point that we could not keep a small number of i/o intensive VMs up for 1 hour and had trouble getting them off. We started troubleshooting the issue a few weeks ago, and finally came to a total resolution early this week. Here is a summary of the issues, how they came to be and how we resolved them:

* http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=226424 (A filer may exhibit poor performance due to WAFL holding on to too many network
buffers and not releasing them in a timely fashion.)
To fix this, we had to do an upgrade to 7.2.4 – that has been completed.

* NetApp LUN’s were created of the wrong LUN type.
This was caused by a error in the LUN creation workflow causing the LUN to be set to the default value (Solaris). 3 out of 4 LUNs were of type Solaris causing blocks to not be written efficiently to the disk (the 4k VMWare blocks were written offset to the true disk geometry). Reading of the LUN would cause many read aheads and at times overwhelm the filer due to the inefficient layout on disk. To remedy this we migrated data off, re-created all of the LUNs and re-migrated the data back.

* NetApp igroup’s set to the wrong type.
Initially Netapp advised that linux igroup type (what maps the LUN to various hosts) were OK for use with VMWare. This was incorrect causing improper scsi reservations and iscsi timeouts. NetApp is updating their internal documentation to reflect this change.

* Network setup issues
Initial setup from NetApp advised us to setup the network in a specific configuration (one link to each upstream switch with a virtual interface bonding them). After further investigation, I found this is *not* the best practice and in fact causing issues with dead HBA paths. To correct this temporarily, we disabled one of the links, having single uplinks (still with redundant heads)

All of these issues created major performance degradation and block level access/corruption problems. They have all been resolved at this point. We still need to adjust the network interfaces to be more redundant.

Special thanks to the release engineering team has been *incredibly* patient with us as we worked through this. I know how frustrating it was and they kept a smile (well, kind of) through the situation – really helped us keep pushing forward to a solution. Thanks also to mrz for the amazing amount of work he put into this…very dedicated to finding a solution no matter what time it was.

To mitigate this issue going forward, we have make two changes.

• Modified the port-channels connecting the core switches to downstream switches to better handle a port-channel member failure.
• We also further tuned broadcast storm protection on every switch port to limit the amount of broadcast & multicast traffic any one device is allowed to send.

Furthermore, we have a priority case open with the vendor to determine the cause of the issue as we did capture debug logs. This was in no way related to the scheduled downtime we were in, it just happened to coincide. We apologize for any inconvenience this may have caused. We’ll continue to follow up with the vendor to make sure this issue does not happen again.

We’d like to build up our mirror network to be even stronger! I am making a call to the community to help us find other mirror sources. Already Paul Vixie from the Internet Software Consortium has stepped up and donated 3gb/s of mirror peak capacity (!). Details on what is required can be found here: http://www.mozilla.org/mirroring.html. While we are always happy to take any mirror donation, we are specifically looking for mirrors which can handle in excess of 100mb/s during peak traffic times. Please contact me directly if you have any ideas of people/organizations/companies that might be willing to donate either bandwidth or mirror space.

So, Schrep asked me to kick off a project to address some of the issues we have with Bugzilla and really invest some time and effort to improve Bugzilla for Mozilla, and the rest of the community. I’ve started by rounding up an initial set of improvements after talking to some of the heavy users within Mozilla, asking for their top complaints and suggestions to improve efficiency in using Bugzilla. Here is what I have come up with.

They gave me plenty of things to work on, but I wanted to open it up to others. I’ve added a section to the bottom of the wiki asking for suggestions – please keep your edits there. If you want to vote up another’s suggestion, just add a +1 to their line. I’ll take the top suggestions/defects and add them into the schedule. Keep in mind we won’t be able to do everything, and are limited in terms of capacity but we are throwing some full time weight behind this to help get this moving.

All our changes are planned to first be applied to BMO, then ported to Bugzilla trunk, so all the code will show up in upstream versions of Bugzilla. We hope to make a difference and move Bugzilla forward in ease of use, performance and innovation.

On a side note, I am looking for community Bugzilla members to help – if you are a Bugzilla developer or know someone who would be willing to help, we’ll take all the help we can get! Contact me at justin at mozilla dot com.

Now let me preface this by saying I am in the outer section of the city in a tech park, and haven’t had time to go into the heart of the city (which I hope to do). But on the 12 hour (!) plane ride over, I had this notion that coming to China would be extremely exotic with very different ways of doing things.

Sure, the Internet access is not the best (i.e Great Firewall, international congestion, etc), food can be…adventurous (chicken neck, frog, snail, turtle, donkey, and others were all on the menu at tonight’s restaurant), the weather & pollution aren’t the best, politics aren’t in line with what I’d vote for, but all in all – it’s just a city, and a great one at that. People eat and hang out a lot, get work done in similar fashions and live their lives.

I think the differences in how people work, live, and interact in different cultures is incredibly interesting – hence why I think I am enjoying my time here so much. The trip has really highlighted that while there are a lot of differences in the way we choose to live, we often forget just how similar we all are

More technical (read: nerdy) posts later on the Great Firewall, Internet access, colo’s, and more.