• 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 529600). Duration 30 minutes.
• 9:00pm PST (0500 UTC) getpersonas.com GLB DNS change. We’ll be making DNS changes to move getpersonas.com to our Zeus GLB servers (bug 525908). No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

* 6:00am PST (1400 UTC) stage.mozilla.org – core operating system will be upgraded from RHEL4 to RHEL5.  Expected downtime 60 minutes, things may not behave quite right for an hour or two after the upgrade as the loose ends get cleaned up.  This will affect any build systems that upload to stage.mozilla.org. (bug 524794)

Feel free to comment directly on the bug or on #build on irc.mozilla.org
if you see issues past the planned downtime.

* 6:00pm PST (0200 UTC) bugzilla.mozilla.org update.  We’ll be
upgrading Bugzilla from version 3.2.4 to 3.4.3 (bug 503055).  Duration 90 minutes.

Feel free to comment directly if you see issues past the planned  downtime.

• 5:00pm PST (0100 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 528272). Duration 30 minutes.
• 7:00pm PST (0300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug 527792). Duration 10 minutes.
• 7:00pm PST (0300 UTC) crash-stats.mozilla.com database maintenance. We’ll be working on the database and will impact the materialized views on crash-stats.mozilla.com. All of the cron generated reports will be down for the duration of this window. See bug 526624 for details. Duration 7 hours.
• 9:00pm PST (0500 UTC) dev-vmware01 ESX server upgrade. We’ll be upgrading the server hardware for dev-vmware01. During this window the following VMs will be offline for 2 hours:
  

  • mrapp-stage03
  • sm-ctc01
  • sm-chat01
  • sm-cms01
  • sm-crm01
  • sm-dekistage01

• 9:00pm PST (0500 UTC) Netscaler OS upgrade. We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

Mozilla uses redundant Cisco Firewall Service Modules in the core Cisco 6509 switches. At about 5:07am the primary unit failed and appears to have crashed within the failover code. This caused an incomplete failover and the standby FWSM never assumed full control.

At 5:19am PST the primary FWSM was manually reset which allowed the standby to complete its failover.

We have grabbed crashinfo and are working with Cisco TAC to diagnose. Specific details will be tracked in bug 528186.

We apologize for any inconvenience this may have caused and will continue to work with Cisco on a remedy.

• 7:00pm PST (0300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 526284) . Duration 60 minutes.
• 7:00pm PST (0300 UTC) getpersonas.com update and database upgrade. We’ll be updating getpersonas.com to pick up code updates (bug 526291) and moving its database to a different database cluster (bug 521597). Duration 10 minutes.
• 7:00pm PST (0300 UTC) smtp.mozilla.org configuration change. We’ll be removing mozilla.com and mozilla.org from the authorized recipients list on the inbound mail servers, since they are no longer used as the MX for those domains (bug 524804) No downtime expected.
• 11:00pm PST (0700 UTC) www.mozilla-europe.org GLB DNS change. We’ll be making DNS changes to move www.mozilla-europe.org to our Zeus GLB servers (bug 523711). No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) labs.mozilla.com migration. We’ll be migrating labs.mozilla.com to mozillalabs.com (bug 489638). Duration 60 minutes.
• 9:00pm PDT (0400 UTC) addons.mozilla.org backend server maintenance. We’ll be making changes to the backend storage (bug 517015) and database servers (bug 524011). Duration 90 minutes.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

* 5:00pm PDT (0000 UTC) addons.mozilla.org database hardware upgrade. We’ll be adding additional RAM to the master database server for addons.mozilla.org. (bug 522112). Duration 30 minutes (although the site may stay down pending completion of the file moves, see below)

* 5:00pm PDT (0000 UTC) We’ll be taking advantage of the database outage to remap part of the back-end filesystem on addons.mozilla.org. (bug 517015).  Duration 90 minutes.

* 7:00pm PDT (0200 UTC) We’ll be making DNS changes to move the support.mozilla.org domain to our Zeus GLB servers. (bug 522923) No downtime expected.

Please let me know if you have any reason why we should not proceed  with this planned maintenance. As always, we aim to keep downtime to  as little as possible, but unexpected complications can arise causing  longer downtime periods than expected. All systems should be  operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned  downtime.

• 9:00pm PDT (0400 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 523428). Duration 30 minutes.
• 9:00pm PDT (0400 UTC) We’ll be rolling changes to push several production websites out of Amsterdam. This involves changing DNS over to our Zeus GLB servers. No downtime expected.
  The following sites will be moved tonight:
  

  • www.spreadfirefox.com
  • support.mozilla.com (bug 522923)

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

This will begin at 11am PDT (about 10 minutes from now)

The following websites will be affected by this outage:

addons.mozilla.org (forums only)
blog.lizardwrangler.com
blog.mozilla.com
blog.mozillafoundation.org
byob.mozilla.com
communitystore.mozilla.org
contribute.mozilla.org
creative.mozilla.org
getfirebug.com
getpersonas.com
hacks.mozilla.org
impact.mozilla.com
kubla.mozilla.com
kubla.mozillaonline.com
labs.mozilla.org
library.mozilla.org
mozillaparty.com
mozillaservice.org
operationfirefox.com
pastebin.mozilla.org
priorartshare.org
quality.mozilla.org
reporter.mozilla.org
survey.mozilla.com
survey.mozilla-europe.org
tools.mozilla.com
userpanel.mozilla.com
warnings.build.mozilla.org

We apologize for any inconvenience, and appreciate your understanding.