• 7:00pm PST (0300 UTC) duration 1 hour: Firmware upgrade. We’ll be upgrading the firmware of the machine that hosts tinderbox.mozilla.org and bonsai.mozilla.org.
• 8:00pm PST (0400 UTC) duration 3 hours: RHEL5 upgrades. We’ll be doing some more RHEL5 upgrades on several machines. This will internally affect DHCP and DNS throughout the MPT network.
• 9:00pm PST (0500 UTC) crash-stats.mozilla.com database maintenance. We’ll be copying the entire Breakpad database in order to set up a replication environment. No downtime is expected, however degraded performance is expected for several days while the database is copied.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in this blog if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) mozilla.org MX change.  We’ll be updating mozilla.org MX records.  No downtime is expected.
• 7:00pm PST (0300 UTC) duration 4 hours: Kernel upgrades.  We’ll be doing kernel security updates on various machines affecting the following services for approximately 10 minutes each:

build graphs / pageload tests
CVS, cvs-mirror, SVN, Hg
LXR/MXR
Despot, Doctor
ftp.mozilla.org
quality.mozilla.org
videos.mozilla.org
stage.mozilla.org
people.mozilla.com
Buildbot Try Server
MDC Search
Internet-facing mail relays, list servers, and outbound mail (mail will be queued)
Any sites using LDAP for authentication
DNS/DHCP/wifi/telephone services in all Mozilla offices (except Beijing)
support.mozilla.com chat server
irc.mozilla.org
All database-backed websites and services, including but not limited to: Bugzilla, Breakpad, Bonsai, SUMO, SpreadFirefox, Addons, bouncer/download, Blogs, QMO, Litmus

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in this blog if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) addons.mozilla.org updates. We’ll be updating addons.mozilla.org to pick up code updates. See bug 469210 for more details. No downtime expected.
• 7:00pm PST (0300 UTC) Bouncer/download.mozilla.org database upgrade. We’ll be upgrading the hardware the Bouncer/download.mozilla.org database runs on. See bug 464748 for more details. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bug (or this blog) if you see issues past the planned downtime.

• 9:00pm PST (0500 UTC) addons.mozilla.org/Zeus ZXTM performance testing. We will be running a performance test of addons.mozilla.org behind a Zeus ZXTM cluster. See bug 467502 for more information (and this blog post).  No downtime is expected, however the test period is expected to run for eighteen (18) hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 10:00pm PST (0600 UTC) border1 Sup720-3BXL upgrade. See http://blog.mozilla.com/mrz/ for more details (this is the second half of Tuesday’s upgrade). During this time all all BGP sessions on border1 will be shutdown and traffic will continue to route through border2. No user-facing downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 6:00pm PST (0200 UTC) bugzilla.mozilla.org updates. We’ll be updating bugzilla.mozilla.org to pick up code updates. See bug 464713 for more details. Duration 5 minutes.
• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 10:00pm PST (0600 UTC) border2 Sup720-3BXL upgrade. See http://blog.mozilla.com/mrz/ for more details. During this time all all BGP sessions on border2 will be shutdown and traffic will continue to route through border1. No user-facing downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 465599. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime. ]]> http://blog.mozilla.com/it/2008/11/18/mozilla-scheduled-downtime-11182008-6pm-12pm-pst-0200-0800-11192008-utc/feed/ http://blog.mozilla.com/it/2008/11/13/mozilla-scheduled-downtime-11132008-7pm-11pm-pst-0300-0700-10242008-utc/ http://blog.mozilla.com/it/2008/11/13/mozilla-scheduled-downtime-11132008-7pm-11pm-pst-0300-0700-10242008-utc/#comments Thu, 13 Nov 2008 23:41:36 +0000 mrz http://blog.mozilla.com/it/?p=194 We will have a scheduled maintenance window tonight from 7:00pm to 11:00pm PST. This is being tracked in master bug 464743. The following changes will take place:

• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PST (0300 UTC) Root-level DNS changes.  We will be updating root-level DNS for mozilla.org, mozilla.com, and spreadfirefox.com to Mozilla-operated nameservers. No downtime is expected.
• 7:00pm PST (0300 UTC) mozilla.com MX change.  We’ll be updating mozilla.com MX records as per bug 464745.  No downtime is expected.
• 7:00pm PST (0300 UTC) Netscaler OS upgrade.  We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes.  No downtime is expected.
• 10:00pm PST (0600 UTC) Cisco FWSM OS upgrade.  We’ll be updating the Cisco firewalls to resolve bug CSCsd67334. The two FWSMs run as an HA pair and this upgrade should be hitless.  No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 464743. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

We will have an emergency maintenance window tonight from 6pm to 11:00pm PST.

The following work will take place:

• 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.  See bug 464458 for details.  We’ll be locking the tree to all check-ins at 5:00pm PST tonight to start the maintenance window and will unlock it once the machines have come back up and gone green for a cycle.

Last night one of our network attached storage devices experienced an error and didn’t fail over properly.  The effect was such that we had to close the tree as Tinderbox wasn’t getting proper data for updates, and we couldn’t tell what was happening with our builds. While we were able to restore that functionality, there is a high chance that any of our VMs attached to this storage device may experience further problems.

To fix this, we’ll need to pause all of our VM systems (most notably build machines & unit test machines, but also litmus and some others) for a few hours. The affected systems would be:

fxdbug-win32-tbox
fx-win32-1.9-slave2
moz2-linux-slave1
patrocles
fx-linux-1.9-slave09
try-master
try-unit-linux-01
xr-linux-tbox
egg
fxdbug-linux-tbox
fx-linux-1.9-slave2
moz2-linux64-slave01
l10n-linux-tbox
prometheus-vm
tb-linux-tbox
try1-win32-slave
try-unit-linux-02
bm-l10n-win2k3-01
dm-bugs-test-app01
dm-graphs-stage01
dm-litmus02
dm-mailman01
fx-win32-1.9-slave08
fx-win32-1.9-slave09
moz2-linux-slave04
moz2-win32-slave04
pm-app01
sand
sm-summit01
moz2-linux-slave09
moz2-linux-slave10
moz2-linux-slave11
moz2-linux-slave12
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave12
moz2-win32-slave13
test-linslave
test-mgmt
moz2-linux-experimental1
moz2-linux-slave01
moz2-linux-slave13
moz2-linux-slave14
moz2-linux-slave15
moz2-linux-slave16
moz2-win32-slave14
moz2-win32-slave15
moz2-win32-slave16
moz2-win32-slave17
moz2-win32-slave18
try-unit-win32-01
test-winslave
try-linux-slave03
try-win32-slave03
bm-buildgraph01
bm-symbolfetch01
dm-ausstage01
dm-chat01
geodns01.sj
dm-graphs01
dm-webtools01
im-bes01
mrapp-intranet01
mrapp-stage01
pm-ns03
production-1.9-master
qm-evtest01
tm-amo01-webdev01

After discussing this with the sheriff, product leads and release engineering team, we’ve decided it would be best to do this as soon as possible. We’ll be locking the tree to all checkins at 5:00pm PST tonight to start the maintenance window, and we’ll unlock it once the machines have come back up and gone green for a cycle. Vlad will try to shepherd the remaining patches for beta 2 blockers that are in the checkin queue before 5pm, but at that time we’ll stop all work and pick it up tomorrow morning.

Let us know if there are any objections as soon as possible.

• 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be updating developer.mozilla.org to pick up code updates. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 460826. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0200 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PDT (0200 UTC) Mailman / mailing list upgrades. We’ll be upgrading Mailman to 2.1.11 and upgrading the host OS to RHEL5. This will affect all mailing lists and all email services under mail.mozilla.org, mozilla.org and bugzilla.org. Duration 3 hours.
• 7:30pm PDT (0230 UTC) mail.mozilla.com / Zimbra Upgrade. We’ll be upgrading Zimbra to 5.0.10. Duration 3-4 hours.
• 8:00pm PDT (0300 UTC) ) MDC / developer.mozilla.org updates. We’ll be moving developer.mozilla.org completely behind SSL. No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 460825. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.