• Migrate from San Jose to Phoenix
• Implement fault tolerant infrastructure

During the maintenance period subversion will be unavailable for both reading and writing. Because we’re
switching to a newer version of subversion (and changing the data store to fsfs) the data migration will require a time consuming svnadmin dump / svnadmin load. We anticipate this step alone will take about 4 hours.

Time: January 28th 8pm-2am PST
Scheduled downtime: 6 hours
Estimated actual downtime: 4.5 hours
Impact: All subversion related services will be unavailable including viewvc

Generally speaking, our datacenter architectures are intentionally simple and spanning tree has served us well. However, as we have grown to meet demand, some of our more… venerable datacenters have become convoluted as new applications are shoehorned into old infrastructure.

Two weeks ago, we brought a new expansion online in Phoenix. Little did we suspect this would be the straw which broke the camel’s back. Minor spanning tree events which had previously gone unnoticed quickly escalated into very noticeable spanning tree cascades. Frustratingly, outages would often resolve themselves before netops personnel could log in to diagnose them. Cell phones vibrated at odd hours of the night. Unkind words were spoken.

Ultimately, we traced the fragility to an oversight in our spanning tree design. Although Juniper is our vendor of choice, we do rely on Cisco’s 3120 blade switch for our HP c7000 chassis. This multi-vendor network creates interesting challenges. In this case, we discovered Juniper’s VSTP mode is not entirely compatible with Cisco’s rapid-pvst mode. In JUNOS versions prior to 10.3, VSTP is unable to fully converge with rapid-pvst. For more information, see Juniper KB 18291 (Juniper support account required).

What did we learn?

1. Be diligent about marking server trunk ports as spanning tree edge ports. Otherwise, these ports will generate topology changes when a server reboots.
2. There’s no such thing as too much logging. Logging of spanning tree events can alert you to unexpected topology changes (See #1).
3. Not all spanning tree protocols are created equal. Don’t blindly trust that spanning tree is doing the right thing.

How do we avoid this, moving forward?

We’re taking great pains to eliminate spanning tree entirely from our newest datacenter, SCL3. While we’re not quite ready to make the leap to a unified fabric architecture (such as Juniper’s QFabric or Cisco’s Nexus), modern multi-chassis technologies can still offer significant improvements. In our case, we’ll be deploying Juniper’s XRE line to enable virtual chassis support on our core EX8200 platform.

Juniper's XRE200

With virtual chassis at every level (core, aggregation, access), we no longer depend on spanning tree for layer 2 redundancy. Instead, we will be able to rely on a link aggregation protocol (such as LACP). This comes with several added benefits:

• Improved utilization and load balancing of redundant links
• Faster convergence
• Capacity for growth
• Not spanning tree

Once this architecture is vetted in SCL3, retrofitting PHX1 with the XRE devices will become a top priority.

The running configuration had the VPN as the active path and the p2p disabled because of an ongoing issue (bug 680463).

Because this path was disabled a complete outage was experienced to SCL1 and all its services which primarily includes the release engineering and build infrastructure.

Upon initial investigation the VPNs, fw1.scl1 and vpn1.sjc1, showed the other was sending a incorrect response while renegotiating the tunnel. Standard non-destructive troubleshooting was attempted to reestablish the tunnel with no success.

In the normal course of troubleshooting fw1.scl1 became unresponsive where on-site presence was required. Once on site fw1.scl1 was restored traffic was shifted from the VPN to the p2p despite it not being confirmed fixed. Basic steps were made to reseat optics and clean fiber patches before traffic was moved.

The review of the logs available did not point to any specific issue why the VPN failed nor why the methods used to recover it failed.

While traffic was being shifted to the p2p the VPN recovered on its own, but the decision was made to stay on the p2p while closely monitoring it being mindful of bug 680463 which has since been resolved.

Netops is investigating configurations to augment link fault and the automatic failover to the standby path and will implement it at a later date.

Complete timeline:

13:24 Initial nagios alert.
13:34 Netops is paged.
13:56 Netops responds.
14:05 Escalation to dmoore (page)
14:13 Escalation to dmoore (phone call)
14:15 Escalation to ravi (page)
14:16 Escalation to ravi (page)
14:16 Ravi responds
14:48 fw1.scl1 becomes unresponsive
16:17 Nagios alerts begin to clear

Tuesday July 12 2011 1800 PST (0100 UTC) [Tree Closure] Duration: 3h

All Firefox, Fennec & TryServer trees will be closed during this window.

The VPN concentrator in SJC1 will be reconfigured to accommodate point-to-point (p2p) links to PHX1 and SCL1.  During this window the p2p to SCL1 will be enabled.

The expected client impact is to Firefox Sync users who change their password. During this time LDAP replication will be suspended and affected users may experience client issues until the replication chain is restored.

Thursday July 14 2011 1800 PST (0100 UTC) Duration: 3h
Tuesday July 19 2011 1800 PST (0100 UTC) Duration: 3h

No tree closure.

Each of these days the router OS will be upgraded on a core and border pair in PHX1.  The upgrades will address bug fixes to allow IPv6 and allow the p2p form SJC1 to be enabled.

There is no expected user facing outage during this work.

Please let me know if you have any reason why we should not proceed with any of the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned windows.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 6:00am PDT (1300 UTC) Talos/Pageloader update.
  RelEng will be landing a changeset on the build/tools infra. This will require restarting all the masters, so we’ll close the tree at 6am, let any running builds complete, then land the patch and restart. Total expected tree closure is 4 hours.
• 9:00am PDT (1600 UTC) SCL1 Switch reconfiguration.
  We will be enabling portfast on the SCL1 top-of-rack switches to improve remote management. Latency may spike on connections to buildslaves, but no loss of connectivity is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 6:00am PDT (1300 UTC) Talos/Pageloader update.
  A new Talos/Pageloader bundle will be deployed. This is expected to cause a wobble in test results, so we’ll be closing the tree to push out the update.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 4:00pm PST (0000 UTC) support.mozilla.com. We will be switching over from the San Jose NFS cluster to the Phoenix NFS cluster (bug 639844). Duration 1 hour.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

The following sites/services may have experienced degraded performance or partial/full outages:

• Firefox Sync
• Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com)
• input.mozilla.com
• pulse.mozilla.org
• firefoxlive.mozilla.org
• demos.mozilla.org
• www.mozillademos.org
• www.drumbeat.org

Background:
There were two separate issues that we encountered, both tracked in bug 639745.

1. DHCP relay failures. This is a regression in the JunOS code.

   Just before 10:00pm Monday night, multiple hosts in Phoenix began to lose their DHCP leases and drop offline. For reasons not yet understood, the DHCP relay feature was no longer operational.

   This caused an 8 minute outage for support.mozilla.com.

2. High CPU load. We began experiencing high (maximum) CPU usage on multiple FPCs after upgrading from 10.1 to 10.4R2. This did not have any immediate impact and we opted to continue working with overnight with JTAC on resolution.

   This morning as general load increased, this became a service impacting issue. Netops downgraded to 10.3R2.11 and eventually downgraded to 10.2S7 to resolve these issues.

We apologize for any inconvenience this may have caused and will continue to work with Juniper to understand why this failed and on a long term remedy.

• 6:00pm PST (0100 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The nature of this upgrade requires redundancy to be disabled and as such, all sites will experience a 10-15 second outage. While the actual user-facing outage is less than a minute, the entire window will be 2 hours.

  The following services will be impacted during this upgrade:
  

  • Firefox Sync
  • Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com)
  • input.mozilla.com
  • pulse.mozilla.org
  • firefoxlive.mozilla.org
  • demos.mozilla.org
  • www.mozillademos.org
  • www.drumbeat.org

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 7:00am PST (1400 UTC) Network hardware reconfiguration.
  Edge switches in one of our datacenters need reconfiguration. This will close all trees, as Talos machines will experience intermittent connectivity during the change. Apologies for the short notice; we want to get this done during a gap in the beta build cycle.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

UPDATE: 08:10 PST
All maintenance completed successfully, and the trees are reopened.