• 5:00pm PDT (0000 UTC) We will be upgrading all the components of Socorro, and we expect this to take up to two hours, less if all goes well.  Web app upgrades (crash-stats.mozilla.com) are minor so user facing downtime should be only a few minutes towards the end of the window. Duration: 2 hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) developer.mozilla.org update. We’ll be updating developer.mozilla.org to pick up code updates. (bug 589248) No downtime expected, but may take up to 2 hours.
• 7:00pm PDT (0200 UTC) support.mozilla.com update.  We’ll be updating support.mozilla.com to pick up code updates. (bug 589861) Duration 2 hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 4:00pm PDT (2300 UTC) support.mozilla.com update. We’ll be updating support.mozilla.com to pick up code updates (bug 587824) . Duration 60 minutes.
• 6:00pm PDT (0100 UTC) Kernel upgrades. We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
  • *.stage.mozilla.com staging websites
  • Mountain View office VPN server
  • people.mozilla.com
  • bugzilla.mozilla.org
  • developer.mozilla.org

  Some services may have more than one outage as various back-end components each get their upgrades (for example, a web application will go down once when the web servers are upgraded and again when its database server is upgraded).

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 6:00pm PDT (0100 UTC) Kernel upgrades. We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
  • *ALL* sm-* (development) servers.
  • All VPNs and jumphosts in the offices and colos (except Beijing)
  • All office and colo DNS and DHCP servers (except Beijing)
  • All Web-dev and app staging servers
  • support.mozilla.com (chat services and knowledge base)
  • people.mozilla.com
  • intranet.mozilla.org
  • quality.mozilla.org
  • irc.mozilla.org (multiple servers, will fail over via DNS to keep it up)
  • download.mozilla.org
  • videos.mozilla.org
  • nagios.mozilla.org
  • bugzilla.mozilla.org
  • spreadfirefox.com
  • Internet-facing mail relays, list servers, and outbound mail (mail will be queued)
  • All public and private wikis and blogs administered by Mozilla

Some services may have more than one outage as various back-end components each get their upgrades (for example, a web application will go down once when the web servers are upgraded and again when its database server is upgraded).

This is a pared down list of the same systems we listed this last Tuesday night, we didn’t manage to get all of them and will be completing the rest tonight.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing   longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 4:00pm PDT (2300 UTC) addons.mozilla.org update. We’ll be updating addons.mozilla.org to pick up code updates (bug 580283) . Duration 60 minutes.
• 9:00pm PDT (0400 UTC) We’ll be upgrading the FWSM firewalls in all of our colo facilities.  The operate in redundant pairs, and will be failed over after insuring the upgraded one is functioning properly, so no downtime is expected. Duration 2 hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm – Upgrade the server ‘mradm01′ from RHEL4 to RHEL5 (bug 542454).  Actual downtime is expected to be 30 minutes or less, but it may take through the end of the window to get all services completely functional again afterwards.  This server hosts ns1.mozilla.org (DNS server) and the nagios slave (facility-wide system monitoring) for the MPT colo facility.  DNS services are redundant (ns2 and ns3 will pick up the slack).  We will be without monitoring in the MPT colo facility while the server is being upgraded, however.  This means any problems that are normally automatically detected and cause the oncall sysadmin to get paged will not be detected or paged about during the outage.  We ask that if anyone observes any such problems during the window that you report them in the #bmo channel on irc.mozilla.org (click the link to go there via your browser).

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm – Breakpad maintenance – We will be upgrading crash-stats.mozilla.org and doing maintenance on the Breakpad database to improve future performance.  crash-stats.mozilla.org will be down for around 4 hours. No crash reports will be processed during this window. We will still be accepting and storing incoming crashes. (bug 542390)

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm Mozilla Developer Center upgrade – We will be upgrading developer.mozilla.org to Mindtouch 9.12.  30 minutes downtime expected. (bug 531977)
• 7:00pm Getfirebug.com upgrade – We will be deploying a site redesign for the getfirebug.com website. 30 minutes downtime expected. (bug 538320)
• 7:00pm jetpackgallery.mozillalabs.com – We will be upgrading the site to version 1.2 of Jetpack Gallery.  15 minutes downtime expected.  (bug 542083)
• 7:30pm Kernel upgrades. We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:
  • breakpad.mozilla.org
  • crash-reports.mozilla.org

  Some services may have more than one outage as various back-end components each get their upgrades (for example, a web application will go down once when the web servers are upgraded and again when its database server is upgraded).

• 8:00pm crash-reports.mozilla.org – We will be performing database maintenance to improve search capability on crash-reports.mozilla.org.  No downtime is expected, but site performance may be degraded for up to 3 hours. (bug 538313)

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

* 6:00am PST (1400 UTC) stage.mozilla.org – core operating system will be upgraded from RHEL4 to RHEL5.  Expected downtime 60 minutes, things may not behave quite right for an hour or two after the upgrade as the loose ends get cleaned up.  This will affect any build systems that upload to stage.mozilla.org. (bug 524794)

Feel free to comment directly on the bug or on #build on irc.mozilla.org
if you see issues past the planned downtime.

* 6:00pm PST (0200 UTC) bugzilla.mozilla.org update.  We’ll be
upgrading Bugzilla from version 3.2.4 to 3.4.3 (bug 503055).  Duration 90 minutes.

Feel free to comment directly if you see issues past the planned  downtime.