• 9:00pm PST (0500 UTC) addons.mozilla.org/Zeus ZXTM performance testing. We will be running a performance test of addons.mozilla.org behind a Zeus ZXTM cluster. See bug 467502 for more information (and this blog post).  No downtime is expected, however the test period is expected to run for eighteen (18) hours.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 10:00pm PST (0600 UTC) border1 Sup720-3BXL upgrade. See http://blog.mozilla.com/mrz/ for more details (this is the second half of Tuesday’s upgrade). During this time all all BGP sessions on border1 will be shutdown and traffic will continue to route through border2. No user-facing downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) support.mozilla.com updates. We’ll be updating support.mozilla.com to pick up code updates. No downtime is expected.
• 7:00pm PST (0300 UTC) Root-level DNS changes.  We will be updating root-level DNS for mozilla.org, mozilla.com, and spreadfirefox.com to Mozilla-operated nameservers. No downtime is expected.
• 7:00pm PST (0300 UTC) mozilla.com MX change.  We’ll be updating mozilla.com MX records as per bug 464745.  No downtime is expected.
• 7:00pm PST (0300 UTC) Netscaler OS upgrade.  We’ll be upgrading the Netscaler load balancers in San Jose, Amsterdam and Beijing to pick up code fixes.  No downtime is expected.
• 10:00pm PST (0600 UTC) Cisco FWSM OS upgrade.  We’ll be updating the Cisco firewalls to resolve bug CSCsd67334. The two FWSMs run as an HA pair and this upgrade should be hitless.  No downtime is expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

As mentioned, this maintenance window is tracked in bug 464743. Feel free to comment directly in that bug (or this blog) if you see issues past the planned downtime.

We will have an emergency maintenance window tonight from 6pm to 11:00pm PST.

The following work will take place:

• 6:00pm PST (0200 UTC) ) EqualLogic storage array firmware upgrade.  See bug 464458 for details.  We’ll be locking the tree to all check-ins at 5:00pm PST tonight to start the maintenance window and will unlock it once the machines have come back up and gone green for a cycle.

Last night one of our network attached storage devices experienced an error and didn’t fail over properly.  The effect was such that we had to close the tree as Tinderbox wasn’t getting proper data for updates, and we couldn’t tell what was happening with our builds. While we were able to restore that functionality, there is a high chance that any of our VMs attached to this storage device may experience further problems.

To fix this, we’ll need to pause all of our VM systems (most notably build machines & unit test machines, but also litmus and some others) for a few hours. The affected systems would be:

fxdbug-win32-tbox
fx-win32-1.9-slave2
moz2-linux-slave1
patrocles
fx-linux-1.9-slave09
try-master
try-unit-linux-01
xr-linux-tbox
egg
fxdbug-linux-tbox
fx-linux-1.9-slave2
moz2-linux64-slave01
l10n-linux-tbox
prometheus-vm
tb-linux-tbox
try1-win32-slave
try-unit-linux-02
bm-l10n-win2k3-01
dm-bugs-test-app01
dm-graphs-stage01
dm-litmus02
dm-mailman01
fx-win32-1.9-slave08
fx-win32-1.9-slave09
moz2-linux-slave04
moz2-win32-slave04
pm-app01
sand
sm-summit01
moz2-linux-slave09
moz2-linux-slave10
moz2-linux-slave11
moz2-linux-slave12
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave12
moz2-win32-slave13
test-linslave
test-mgmt
moz2-linux-experimental1
moz2-linux-slave01
moz2-linux-slave13
moz2-linux-slave14
moz2-linux-slave15
moz2-linux-slave16
moz2-win32-slave14
moz2-win32-slave15
moz2-win32-slave16
moz2-win32-slave17
moz2-win32-slave18
try-unit-win32-01
test-winslave
try-linux-slave03
try-win32-slave03
bm-buildgraph01
bm-symbolfetch01
dm-ausstage01
dm-chat01
geodns01.sj
dm-graphs01
dm-webtools01
im-bes01
mrapp-intranet01
mrapp-stage01
pm-ns03
production-1.9-master
qm-evtest01
tm-amo01-webdev01

After discussing this with the sheriff, product leads and release engineering team, we’ve decided it would be best to do this as soon as possible. We’ll be locking the tree to all checkins at 5:00pm PST tonight to start the maintenance window, and we’ll unlock it once the machines have come back up and gone green for a cycle. Vlad will try to shepherd the remaining patches for beta 2 blockers that are in the checkin queue before 5pm, but at that time we’ll stop all work and pick it up tomorrow morning.

Let us know if there are any objections as soon as possible.

We will have a scheduled downtime window this coming Saturday from 10pm to 12pm PST. The following changes will take place:

* 10pm PST - Load balancer software upgrade. We will be updating the code on our primary load balancers to address specific known issues. There have been a issues with previous upgrades, and we have backed them out but we believe we have the solution and this should be the final upgrade attempt. There is no estimated downtime as they are in a fault tolerant configuration, but downtime is possible.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra at mozilla.org if you see issues past the planned downtime.

-Justin

We will have a scheduled downtime window this coming Thursday from 7pm to 10pm PST. The following changes will take place:

* 7pm PST - Load balancer software upgrade. We will be updating the code on our primary load balancers to address specific known issues. There is no estimated downtime as they are in a fault tolerant configuration, but downtime is possible.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin

We will have a scheduled downtime window tonight from 7pm to 10pm PST. The following changes will take place:

* 7pm PST - Website back-end migration. We will be modifying how the code is pushed from our source repo to production. No downtime is expected
* 7pm PST - Bugzilla infrastructure upgrade. We will be adding some support libraries needed for the upcoming bugzilla upgrade. No downtime is expected
* 8pm PST - VPN server reboot. We will be rebooting out San Jose VPN server to update time sync settings. 15 minutes of downtime is expected.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

-Justin

We will have a scheduled downtime window this coming Thursday from 7pm to 11pm PST. The following changes will take place:

* 7pm PST - Netapp shelf firmware upgrade. We will need to take one of our Netapp filers offline for a disk shelf upgrade. Estimated downtime is 30 minutes.
* 7pm PST - Bonsai, Tinderbox and Registry will be migrated to a new server. Estimated downtime is 1 hour.
* 8pm PST - Netapp OS upgrade. We will be upgrading both our filers software for known issues. No downtime is expected as we are in a clustered configuration.
* 8:30pm PST - Mozilla.com failover testing. We will be testing the failover capabilities and troubleshooting a few known issues. Expect intermittent mail access during the window, although no mail will be lost.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

We will have a scheduled downtime window tonight from 7pm to 10pm PDT. The following changes will take place:

* 7pm PDT - Continued database migrations. We have 2 new database servers to help alleviate some of the load. Databases to migrate are: devmowiki, firefoxflicks, labs, quality_wiki, reporter, userpanel, bonsai, partners and survey. Applications may be down or in a read-only state for the duration of the window.

Please let me know if you have any reason why we should not proceed with the planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Please let me know if you have any reason why we should not proceed with the planned maintenance. Feel free to email infra@mozilla.org if you see issues past the planned downtime.

Thanks,
-Justin