Mozilla IT

Mozilla IT & Operations

Archive for Infrastructure Notices

Mozilla Scheduled Maintenance, Subversion (svn.mozilla.org) will be unavailable 01/28/2012 8pm-2am PST (0400 GMT)

Posted by bhourigan

We will have a scheduled maintenance window on Saturday, January 28th at 8pm-2am PST (0400 GMT). The following work will take place: Migrate from San Jose to Phoenix Implement fault tolerant infrastructure During the maintenance period subversion will be unavailable for both reading and writing. Because we’re switching to a newer version of subversion (and… Read more

Tags: , , ,

Categories: General Updates, Outages, Scheduled Maintenance

To Tree or Not to Tree

Posted by dmoore

This week, our Phoenix datacenter fell prey to a series of brief rolling outages which visibly impacted many of Mozilla’s public services. Generally speaking, our datacenter architectures are intentionally simple and spanning tree has served us well. However, as we have grown to meet demand, some of our more… venerable datacenters have become convoluted as… Read more

Tags: , ,

Categories: Outages

RFO: SCL1 outage Oct 16, 2011

Posted by ravi

On October 13th at 1324 PST Nagios alerted the start of a network event affecting reachability to the SCL1 data center. SCL1 is configured with redundant internet links where a VPN traverses a redundant firewall at both ends. There is also a point-to-point (p2p) that connects directly to SJC1. The running configuration had the VPN… Read more

Tags: ,

Categories: Outages

Mozilla Scheduled Maintenance 7/12 (tree closure), 14, 19/2011 1800 PST (0100 UTC)

Posted by ravi

Network Operations will be conducting the following maintenance windows: Tuesday July 12 2011 1800 PST (0100 UTC) [Tree Closure] Duration: 3h All Firefox, Fennec & TryServer trees will be closed during this window. The VPN concentrator in SJC1 will be reconfigured to accommodate point-to-point (p2p) links to PHX1 and SCL1.  During this window the p2p… Read more

Tags: , ,

Categories: Infrastructure Notices, RelEng, Scheduled Maintenance

Mozilla Scheduled Maintenance (Tree Closure) – 5/31/2011, 6am PDT (1300 UTC 31 May 2011)

Posted by amilewski

We will have a scheduled maintenance window Tuesday morning, May 31 from 6am to 10am PDT (1300-1700 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 6:00am PDT (1300 UTC) Talos/Pageloader update. RelEng will be landing a changeset on the build/tools infra. This will require restarting all the masters, so we’ll… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Maintenance (Tree Closure) – 5/9/2011, 6am PDT (1300 UTC 9 May 2011)

Posted by amilewski

We will have a scheduled maintenance window Monday morning, May 9 from 6am to 7am PDT (1300-1400 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 6:00am PDT (1300 UTC) Talos/Pageloader update. A new Talos/Pageloader bundle will be deployed. This is expected to cause a wobble in test results, so we’ll… Read more

Categories: Scheduled Maintenance

Mozilla Network Outage Report (Phoenix) – 03/08/2011, 5:00am PST – 11:30am PST

Posted by mrz

For several hours this morning, Mozilla’s Phoenix data center suffered several intermittent outages. This was fall out from yesterday’s Juniper SRX JunOS upgrade. The following sites/services may have experienced degraded performance or partial/full outages: Firefox Sync Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com) input.mozilla.com pulse.mozilla.org firefoxlive.mozilla.org demos.mozilla.org www.mozillademos.org www.drumbeat.org Background: There were two separate issues that we encountered,… Read more

Tags: ,

Categories: Outages

Mozilla Scheduled Maintenance – 03/07/2011, 6pm PST (03/08/2011, 0100 UTC)

Posted by mrz

We will have an off-schedule maintenance window tomorrow night from 6:00pm to 9:00pm PST. The following changes will take place: 6:00pm PST (0100 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The nature of this upgrade requires redundancy to be disabled and as such, all sites will experience a 10-15 second outage.… Read more

Categories: Scheduled Maintenance

Mozilla Scheduled Maintenance (Tree Closure) – 2/18/2011, 7am PST (1500 UTC 18 Feb 2011)

Posted by amilewski

We will have a scheduled maintenance window tomorrow (Friday) morning from 7am to 8am PST (1500-1600 UTC) All Firefox, Fennec & TryServer trees will be closed during this window. 7:00am PST (1400 UTC) Network hardware reconfiguration. Edge switches in one of our datacenters need reconfiguration. This will close all trees, as Talos machines will experience… Read more

Categories: Scheduled Maintenance