• 6:00pm PST (0200 UTC) bugzilla.mozilla.org updates. We’ll be updating bugzilla.mozilla.org to pick up code updates. See bug 464713 for more details. Duration 5 minutes.
• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 10:00pm PST (0600 UTC) border2 Sup720-3BXL upgrade. See http://blog.mozilla.com/mrz/ for more details. During this time all all BGP sessions on border2 will be shutdown and traffic will continue to route through border1. No user-facing downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 465599. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime. ]]> http://blog.mozilla.com/it/2008/11/18/mozilla-scheduled-downtime-11182008-6pm-12pm-pst-0200-0800-11192008-utc/feed/ http://blog.mozilla.com/it/2008/11/13/mozilla-scheduled-downtime-11132008-7pm-11pm-pst-0300-0700-10242008-utc/ http://blog.mozilla.com/it/2008/11/13/mozilla-scheduled-downtime-11132008-7pm-11pm-pst-0300-0700-10242008-utc/#comments Thu, 13 Nov 2008 23:41:36 +0000 mrz http://blog.mozilla.com/it/?p=194 We will have a scheduled maintenance window tonight from 7:00pm to 11:00pm PST. This is being tracked in master bug 464743. The following changes will take place:

• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PST (0300 UTC) Root-level DNS changes.  We will be updating root-level DNS for mozilla.org, mozilla.com, and spreadfirefox.com to Mozilla-operated nameservers. No downtime is expected.
• 7:00pm PST (0300 UTC) mozilla.com MX change.  We’ll be updating mozilla.com MX records as per bug 464745.  No downtime is expected.
• 7:00pm PST (0300 UTC) Netscaler OS upgrade.  We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes.  No downtime is expected.
• 10:00pm PST (0600 UTC) Cisco FWSM OS upgrade.  We’ll be updating the Cisco firewalls to resolve bug CSCsd67334. The two FWSMs run as an HA pair and this upgrade should be hitless.  No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 464743. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

We will have an emergency maintenance window tonight from 6pm to 11:00pm PST.

The following work will take place:

• 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.  See bug 464458 for details.  We’ll be locking the tree to all check-ins at 5:00pm PST tonight to start the maintenance window and will unlock it once the machines have come back up and gone green for a cycle.

Last night one of our network attached storage devices experienced an error and didn’t fail over properly.  The effect was such that we had to close the tree as Tinderbox wasn’t getting proper data for updates, and we couldn’t tell what was happening with our builds. While we were able to restore that functionality, there is a high chance that any of our VMs attached to this storage device may experience further problems.

To fix this, we’ll need to pause all of our VM systems (most notably build machines & unit test machines, but also litmus and some others) for a few hours. The affected systems would be:

fxdbug-win32-tbox
fx-win32-1.9-slave2
moz2-linux-slave1
patrocles
fx-linux-1.9-slave09
try-master
try-unit-linux-01
xr-linux-tbox
egg
fxdbug-linux-tbox
fx-linux-1.9-slave2
moz2-linux64-slave01
l10n-linux-tbox
prometheus-vm
tb-linux-tbox
try1-win32-slave
try-unit-linux-02
bm-l10n-win2k3-01
dm-bugs-test-app01
dm-graphs-stage01
dm-litmus02
dm-mailman01
fx-win32-1.9-slave08
fx-win32-1.9-slave09
moz2-linux-slave04
moz2-win32-slave04
pm-app01
sand
sm-summit01
moz2-linux-slave09
moz2-linux-slave10
moz2-linux-slave11
moz2-linux-slave12
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave12
moz2-win32-slave13
test-linslave
test-mgmt
moz2-linux-experimental1
moz2-linux-slave01
moz2-linux-slave13
moz2-linux-slave14
moz2-linux-slave15
moz2-linux-slave16
moz2-win32-slave14
moz2-win32-slave15
moz2-win32-slave16
moz2-win32-slave17
moz2-win32-slave18
try-unit-win32-01
test-winslave
try-linux-slave03
try-win32-slave03
bm-buildgraph01
bm-symbolfetch01
dm-ausstage01
dm-chat01
geodns01.sj
dm-graphs01
dm-webtools01
im-bes01
mrapp-intranet01
mrapp-stage01
pm-ns03
production-1.9-master
qm-evtest01
tm-amo01-webdev01

After discussing this with the sheriff, product leads and release engineering team, we’ve decided it would be best to do this as soon as possible. We’ll be locking the tree to all checkins at 5:00pm PST tonight to start the maintenance window, and we’ll unlock it once the machines have come back up and gone green for a cycle. Vlad will try to shepherd the remaining patches for beta 2 blockers that are in the checkin queue before 5pm, but at that time we’ll stop all work and pick it up tomorrow morning.

Let us know if there are any objections as soon as possible.

• 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be updating developer.mozilla.org to pick up code updates. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 460826. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PDT (0200 UTC) Mailman / mailing list upgrades. We’ll be upgrading Mailman to 2.1.11 and upgrading the host OS to RHEL5. This will affect all mailing lists and all email services under mail.mozilla.org, mozilla.org and bugzilla.org. Duration 3 hours.
• 7:30pm PDT (0230 UTC) mail.mozilla.com / Zimbra Upgrade. We’ll be upgrading Zimbra to 5.0.10. Duration 3-4 hours.
• 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be moving developer.mozilla.org completely behind SSL. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 460825. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) Thirdlane PBX upgrades. We will be upgrading the configuration GUI and its underlying Asterisk add-on features for our phone system. The phone system itself in Mountain View will probably be down less than 5 minutes. Specific features (conference call rooms, provisioning of new telephones, calls to phones in Toronto) may be offline for up to 6 hours. Duration 6 hours
• 7:00pm PDT (0200 UTC) Mailman / mailing list upgrades. We’ll be upgrading Mailman to 2.1.11 and upgrading the host OS to RHEL5. This will affect all mailing lists and all email services under lists.mozilla.org, mail.mozilla.org, mozilla.org and bugzilla.org. Duration 3 hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 459262. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) addons.mozilla.org/*.addons.mozilla.org database restart and code updates. We’ll be updating addons.mozilla.org to pick up code updates. We’ll also be restarting the master database to change some configuration settings. See bug 458110 for more details. This will impact addons.mozilla.org and all related sub-domains. Duration 15 minutes.
• 7:00pm PDT (0200 UTC) Final MySQL 5 migration. We will be completing our last MySQL 4 master to MySQL 5. The following services will go offline for up to 30 minutes while they are moved to a new master database:
  • Bonsai
  • Despot
  • Graph Server
  • Litmus
• 9:00pm PDT (0200 UTC) San Jose Netscaler software upgrade. We’ll be doing a vendor recommended upgrade on the San Jose Netscalers. No downtime expected.
• 10:00pm PDT (0200 UTC) Cisco IOS upgrade. We’ll be upgrading IOS on two of the embedded HP BladeSystem Cisco switches to pick up a bug fix (BugID CSCsf17036). The systems attached to these switches are multi-homed. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 458901. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PDT (0200 UTC) spreadfirefox.com updates. We’ll be updating spreadfirefox.com to pick up code updates. No downtime is expected.
• 7:00pm PDT (0200 UTC) Wiki upgrades. We’ll be upgrading all Intranet wikis hosted by Mozilla. This will affect anything under intranet.mozilla.org. Duration 3 hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 456442. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

We need to take our LDAP infrastructure offline for two hours tomorrow night, which will affect everything that uses LDAP for authentication, including Hg and SVN.

We’ll also use that same time to take cvs.mozilla.org offline for maintenance.

During this time, anonymous checkouts via http/https from svn.mozilla.org and hg.mozilla.org will continue to work, as well as pserver checkouts from cvs-mirror.mozilla.org.

Check-ins for SVN will be disabled until LDAP is back online. For Hg, you can still commit locally, but will need to wait for LDAP to return before pushing your commits back to the server. CVS commits will be available when cvs.mozilla.org’s (separate) maintenance is completed.

This is planned for tomorrow night (Sunday) starting at 10pm PDT and lasting until about 12am PDT (0500 - 0700 UTC).

Please let me know if this time frame is entirely inconvenient.

At around 3:30am PDT, Mozilla’s primary datacenter in San Jose went offline.

From initial conversations with CRG West, we learned they had a catastrophic HVAC failure at 2:46am PDT.  CRG West later reported,

“…service interruption that affected customers at the Market Post Tower data center this morning from approximately 2:46am to 5:30am PST.  After thoroughly looking into this matter, CRG West Operations has identified a tripped utility main input breaker feeding the 3rd, 10th and 16th floors of the data center as the cause of the issue.”

At around 3:17am PDT, as a preventative measure, a lot of the servers started to automatically shut themselves down to prevent thermal damage.  Ambient air temperature was somewhere close to 42° C.

Mozilla IT was onsite shortly afterwards and, after dealing with a failed out-of-band/NMS switch and blown fuses, had the majority of the infrastructure back online by 9:10am PDT and declared everything “all clear” by 9:47am PDT.

We apologize for any inconvenience this may have caused and will continue to follow up with CRG West.