• 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 529600). Duration 30 minutes.
• 9:00pm PST (0500 UTC) getpersonas.com GLB DNS change. We’ll be making DNS changes to move getpersonas.com to our Zeus GLB servers (bug 525908). No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 528272). Duration 30 minutes.
• 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug 527792). Duration 10 minutes.
• 7:00pm PST (0300 UTC) crash-stats.mozilla.com database maintenance. We’ll be working on the database and will impact the materialized views on crash-stats.mozilla.com. All of the cron generated reports will be down for the duration of this window. See bug 526624 for details. Duration 7 hours.
• 9:00pm PST (0500 UTC) dev-vmware01 ESX server upgrade. We’ll be upgrading the server hardware for dev-vmware01. During this window the following VMs will be offline for 2 hours:
  

  • mrapp-stage03
  • sm-ctc01
  • sm-chat01
  • sm-cms01
  • sm-crm01
  • sm-dekistage01

• 9:00pm PST (0500 UTC) Netscaler OS upgrade. We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

Mozilla uses redundant Cisco Firewall Service Modules in the core Cisco 6509 switches. At about 5:07am the primary unit failed and appears to have crashed within the failover code. This caused an incomplete failover and the standby FWSM never assumed full control.

At 5:19am PST the primary FWSM was manually reset which allowed the standby to complete its failover.

We have grabbed crashinfo and are working with Cisco TAC to diagnose. Specific details will be tracked in bug 528186.

We apologize for any inconvenience this may have caused and will continue to work with Cisco on a remedy.

• 7:00pm PST (0300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 526284) . Duration 60 minutes.
• 7:00pm PST (0300 UTC) getpersonas.com update and database upgrade. We’ll be updating getpersonas.com to pick up code updates (bug 526291) and moving its database to a different database cluster (bug 521597). Duration 10 minutes.
• 7:00pm PST (0300 UTC) smtp.mozilla.org configuration change. We’ll be removing mozilla.com and mozilla.org from the authorized recipients list on the inbound mail servers, since they are no longer used as the MX for those domains (bug 524804) No downtime expected.
• 11:00pm PST (0700 UTC) www.mozilla-europe.org GLB DNS change. We’ll be making DNS changes to move www.mozilla-europe.org to our Zeus GLB servers (bug 523711). No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) labs.mozilla.com migration. We’ll be migrating labs.mozilla.com to mozillalabs.com (bug 489638). Duration 60 minutes.
• 9:00pm PDT (0400 UTC) addons.mozilla.org backend server maintenance. We’ll be making changes to the backend storage (bug 517015) and database servers (bug 524011). Duration 90 minutes.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 9:00pm PDT (0400 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 523428). Duration 30 minutes.
• 9:00pm PDT (0400 UTC) We’ll be rolling changes to push several production websites out of Amsterdam. This involves changing DNS over to our Zeus GLB servers. No downtime expected.
  The following sites will be moved tonight:
  

  • www.spreadfirefox.com
  • support.mozilla.com (bug 522923)

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:30pm PDT (0230 UTC): Cisco FWSM OS upgrade. We’ll be updating the Cisco firewall module. No downtime is expected.
• 8:00pm PDT (0300 UTC) support.mozilla.com / SUMO updates. We’ll be updating support.mozilla.com to pick up code updates (bug 520042). No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 6:00pm PDT (0200 UTC) Weave server upgrade. Code updates to support the 0.7 release. No downtime expected.
• 7:00pm PDT (0300 UTC) Personas server upgrade. Code updates to enable Firefox 3.6pre builds. Duration 1 hour. Browsers may show a blank persona during this window.
• 9:00pm PDT (0500 UTC) addons.mozilla.org update. We’ll be picking up code updates.. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm PDT (0300 UTC) crash-stats.mozilla.com content push. We’ll be picking up code updates (bug 518636). During this time, crash-stats.mozilla.com should be up but some aggregated reports may not be available. crash-reports.mozilla.com will be up and accept crashes. Duration two hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 6:00pm PDT (0100 UTC) developer.mozilla.org / MDC upgrade. We’ll be updating the software for developer.mozilla.org tonight. See bug 511852 for details. Duration 60 minutes.
• 6:00pm PDT (0100 UTC) SiteSpect deployment. We’ll be implementing multivariate testing with SiteSpect on www.mozilla.com. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bugs if you see issues past the planned downtime.