• Migrate from San Jose to Phoenix
• Implement fault tolerant infrastructure

During the maintenance period subversion will be unavailable for both reading and writing. Because we’re
switching to a newer version of subversion (and changing the data store to fsfs) the data migration will require a time consuming svnadmin dump / svnadmin load. We anticipate this step alone will take about 4 hours.

Time: January 28th 8pm-2am PST
Scheduled downtime: 6 hours
Estimated actual downtime: 4.5 hours
Impact: All subversion related services will be unavailable including viewvc

Tuesday July 12 2011 1800 PST (0100 UTC) [Tree Closure] Duration: 3h

All Firefox, Fennec & TryServer trees will be closed during this window.

The VPN concentrator in SJC1 will be reconfigured to accommodate point-to-point (p2p) links to PHX1 and SCL1.  During this window the p2p to SCL1 will be enabled.

The expected client impact is to Firefox Sync users who change their password. During this time LDAP replication will be suspended and affected users may experience client issues until the replication chain is restored.

Thursday July 14 2011 1800 PST (0100 UTC) Duration: 3h
Tuesday July 19 2011 1800 PST (0100 UTC) Duration: 3h

No tree closure.

Each of these days the router OS will be upgraded on a core and border pair in PHX1.  The upgrades will address bug fixes to allow IPv6 and allow the p2p form SJC1 to be enabled.

There is no expected user facing outage during this work.

Please let me know if you have any reason why we should not proceed with any of the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned windows.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 6:00am PDT (1300 UTC) Talos/Pageloader update.
  RelEng will be landing a changeset on the build/tools infra. This will require restarting all the masters, so we’ll close the tree at 6am, let any running builds complete, then land the patch and restart. Total expected tree closure is 4 hours.
• 9:00am PDT (1600 UTC) SCL1 Switch reconfiguration.
  We will be enabling portfast on the SCL1 top-of-rack switches to improve remote management. Latency may spike on connections to buildslaves, but no loss of connectivity is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 6:00am PDT (1300 UTC) Talos/Pageloader update.
  A new Talos/Pageloader bundle will be deployed. This is expected to cause a wobble in test results, so we’ll be closing the tree to push out the update.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 4:00pm PST (0000 UTC) support.mozilla.com. We will be switching over from the San Jose NFS cluster to the Phoenix NFS cluster (bug 639844). Duration 1 hour.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 6:00pm PST (0100 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The nature of this upgrade requires redundancy to be disabled and as such, all sites will experience a 10-15 second outage. While the actual user-facing outage is less than a minute, the entire window will be 2 hours.

  The following services will be impacted during this upgrade:
  

  • Firefox Sync
  • Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com)
  • input.mozilla.com
  • pulse.mozilla.org
  • firefoxlive.mozilla.org
  • demos.mozilla.org
  • www.mozillademos.org
  • www.drumbeat.org

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 7:00am PST (1400 UTC) Network hardware reconfiguration.
  Edge switches in one of our datacenters need reconfiguration. This will close all trees, as Talos machines will experience intermittent connectivity during the change. Apologies for the short notice; we want to get this done during a gap in the beta build cycle.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

UPDATE: 08:10 PST
All maintenance completed successfully, and the trees are reopened.

All Firefox, Fennec & TryServer trees will be closed during this window.

• 6:00am PST (1400 UTC) Talos Screen Resolution Change.
  Screen Resolutions on Fedora and Fedora64 machines running reftests need to be increased to support reftests with hardware acceleration on Linux.  This is required for bug 593874, and you can follow along in bug 633019.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

UPDATE: 08:00 PST
All maintenance completed successfully, however the tree remains closed due to an unrelated regression.

• 4:00pm PST (0000 UTC) support.mozilla.com primary data center migration. In preparation for Firefox 4, we’ll be moving the primary data center serving support.mozilla.com to Phoenix (bug 622752). During the migration the site will remain online but in a read-only mode. Duration 1 hour.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 3:00pm PST (2300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug 630990). The site will remain up and online during this push, however, user logins will be disabled for the duration of the database update. Duration 2 hours.
• 7:00pm PST (0300 UTC) CVS mirror firmware upgrade. We’ll be applying vendor recommended firmware updates to the servers behind cvs-mirror.mozilla.org which will require a reboot. (Last week’s attempt was aborted.) Duration 15 minutes.
• 9:00pm PST (0500 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The firewalls operate in redundant mode, and will be failed over after insuring the upgraded one is functioning properly. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.