• 7:00pm PST (0300 UTC) OSPF change. We’ll be deploying a configuration change on how the default route from BGP speaking routers is injected into OSPF. See bug 466111 for more detail. No downtime expected.
• 9:00pm PST (0500 UTC) addons.mozilla.org Cisco ACE/Zeus ZXTM performance testing. We will be running a performance test of addons.mozilla.org behind the production ACE/Zeus ZXTM cluster. See bug 477130 for more information. This is the second load test similar to bug 467502 but on the new production hardware. No downtime is expected, however the test period is expected to run for eighteen (18) hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bugs (or this blog) if you see issues past the planned downtime.

• 7:00pm PDT (0300 UTC) crash-stats.mozilla.com (Socorro) database maintenance. We’ll be partitioning the production Socorro database (refer to bug 475815 for details). During this time, crash-stats.mozilla.com will be offline. crash-reports.mozilla.com will be up and accept crashes. They will be queued and processed later. Downtime is expected to last into Saturday but total time is unknown and could run throughout the weekend.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bugs if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) addons.mozilla.org updates. We’ll be updating addons.mozilla.org to pick up code updates (bug 475774). No downtime expected.
• 7:00pm PDT (0300 UTC) ) crash-stats.mozilla.com (Socorro) database maintenance. We’ll be partitioning the production Socorro database (refer to bug 475815 for details). During this time, crash-stats.mozilla.com will be offline. crash-reports.mozilla.com will be up and accept crashes. They will be queued and processed later. Downtime is expected to last into Friday but total time is unknown.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bugs if you see issues past the planned downtime.

• 7:00pm – We will be replacing a faulty midplane on a blade chassis containing 14 servers which support the following services:  addons.mozilla.org, download.mozilla.org, crash-stats.mozilla.com.  With the exception of crash-stats, the affected servers are all redundant or non-production, and therefore no actual downtime is expected.  crash-stats will still be working, however status pages will not update for the duration of the window. (canceled due to schedule conflict, will reschedule at a later date)
• 8:00pm – We will be reverting to a previous version of the DAHDI drivers for our telephone interface cards on the phone server in Mountain View in order to avoid a kernel panic that’s been happening frequently with the current version.  Expected downtime is about 10 minutes.

Please let us know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in those bug (or this blog) if you see issues past the planned downtime.

• 9:00pm PST (0500 UTC) addons.mozilla.org/Zeus ZXTM performance testing. We will be running a performance test of addons.mozilla.org behind a Zeus ZXTM cluster. See bug 467502 for more information (and this blog post).  No downtime is expected, however the test period is expected to run for eighteen (18) hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 10:00pm PST (0600 UTC) border1 Sup720-3BXL upgrade. See http://blog.mozilla.com/mrz/ for more details (this is the second half of Tuesday’s upgrade). During this time all all BGP sessions on border1 will be shutdown and traffic will continue to route through border2. No user-facing downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PST (0300 UTC) Root-level DNS changes.  We will be updating root-level DNS for mozilla.org, mozilla.com, and spreadfirefox.com to Mozilla-operated nameservers. No downtime is expected.
• 7:00pm PST (0300 UTC) mozilla.com MX change.  We’ll be updating mozilla.com MX records as per bug 464745.  No downtime is expected.
• 7:00pm PST (0300 UTC) Netscaler OS upgrade.  We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes.  No downtime is expected.
• 10:00pm PST (0600 UTC) Cisco FWSM OS upgrade.  We’ll be updating the Cisco firewalls to resolve bug CSCsd67334. The two FWSMs run as an HA pair and this upgrade should be hitless.  No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 464743. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

We will have an emergency maintenance window tonight from 6pm to 11:00pm PST.

The following work will take place:

• 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.  See bug 464458 for details.  We’ll be locking the tree to all check-ins at 5:00pm PST tonight to start the maintenance window and will unlock it once the machines have come back up and gone green for a cycle.

Last night one of our network attached storage devices experienced an error and didn’t fail over properly.  The effect was such that we had to close the tree as Tinderbox wasn’t getting proper data for updates, and we couldn’t tell what was happening with our builds. While we were able to restore that functionality, there is a high chance that any of our VMs attached to this storage device may experience further problems.

To fix this, we’ll need to pause all of our VM systems (most notably build machines & unit test machines, but also litmus and some others) for a few hours. The affected systems would be:

fxdbug-win32-tbox
fx-win32-1.9-slave2
moz2-linux-slave1
patrocles
fx-linux-1.9-slave09
try-master
try-unit-linux-01
xr-linux-tbox
egg
fxdbug-linux-tbox
fx-linux-1.9-slave2
moz2-linux64-slave01
l10n-linux-tbox
prometheus-vm
tb-linux-tbox
try1-win32-slave
try-unit-linux-02
bm-l10n-win2k3-01
dm-bugs-test-app01
dm-graphs-stage01
dm-litmus02
dm-mailman01
fx-win32-1.9-slave08
fx-win32-1.9-slave09
moz2-linux-slave04
moz2-win32-slave04
pm-app01
sand
sm-summit01
moz2-linux-slave09
moz2-linux-slave10
moz2-linux-slave11
moz2-linux-slave12
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave12
moz2-win32-slave13
test-linslave
test-mgmt
moz2-linux-experimental1
moz2-linux-slave01
moz2-linux-slave13
moz2-linux-slave14
moz2-linux-slave15
moz2-linux-slave16
moz2-win32-slave14
moz2-win32-slave15
moz2-win32-slave16
moz2-win32-slave17
moz2-win32-slave18
try-unit-win32-01
test-winslave
try-linux-slave03
try-win32-slave03
bm-buildgraph01
bm-symbolfetch01
dm-ausstage01
dm-chat01
geodns01.sj
dm-graphs01
dm-webtools01
im-bes01
mrapp-intranet01
mrapp-stage01
pm-ns03
production-1.9-master
qm-evtest01
tm-amo01-webdev01

After discussing this with the sheriff, product leads and release engineering team, we’ve decided it would be best to do this as soon as possible. We’ll be locking the tree to all checkins at 5:00pm PST tonight to start the maintenance window, and we’ll unlock it once the machines have come back up and gone green for a cycle. Vlad will try to shepherd the remaining patches for beta 2 blockers that are in the checkin queue before 5pm, but at that time we’ll stop all work and pick it up tomorrow morning.

Let us know if there are any objections as soon as possible.

We will have a scheduled downtime window this coming Saturday from 10pm to 12pm PST. The following changes will take place:

* 10pm PST – Load balancer software upgrade. We will be updating the code on our primary load balancers to address specific known issues. There have been a issues with previous upgrades, and we have backed them out but we believe we have the solution and this should be the final upgrade attempt. There is no estimated downtime as they are in a fault tolerant configuration, but downtime is possible.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.org if you see issues past the planned downtime.

-Justin

We will have a scheduled downtime window this coming Thursday from 7pm to 10pm PST. The following changes will take place:

* 7pm PST – Load balancer software upgrade. We will be updating the code on our primary load balancers to address specific known issues. There is no estimated downtime as they are in a fault tolerant configuration, but downtime is possible.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin