Mozilla IT

Mozilla IT & Operations

Archive for Outages

Mozilla Scheduled Maintenance, Subversion (svn.mozilla.org) will be unavailable 01/28/2012 8pm-2am PST (0400 GMT)

Posted by bhourigan

We will have a scheduled maintenance window on Saturday, January 28th at 8pm-2am PST (0400 GMT). The following work will take place: Migrate from San Jose to Phoenix Implement fault tolerant infrastructure During the maintenance period subversion will be unavailable for both reading and writing. Because we’re switching to a newer version of subversion (and… Read more

Tags: , , ,

Categories: General Updates, Outages, Scheduled Maintenance

To Tree or Not to Tree

Posted by dmoore

This week, our Phoenix datacenter fell prey to a series of brief rolling outages which visibly impacted many of Mozilla’s public services. Generally speaking, our datacenter architectures are intentionally simple and spanning tree has served us well. However, as we have grown to meet demand, some of our more… venerable datacenters have become convoluted as… Read more

Tags: , ,

Categories: Outages

RFO: SCL1 outage Oct 16, 2011

Posted by ravi

On October 13th at 1324 PST Nagios alerted the start of a network event affecting reachability to the SCL1 data center. SCL1 is configured with redundant internet links where a VPN traverses a redundant firewall at both ends. There is also a point-to-point (p2p) that connects directly to SJC1. The running configuration had the VPN… Read more

Tags: ,

Categories: Outages

Mozilla Network Outage Report (Phoenix) – 03/08/2011, 5:00am PST – 11:30am PST

Posted by mrz

For several hours this morning, Mozilla’s Phoenix data center suffered several intermittent outages. This was fall out from yesterday’s Juniper SRX JunOS upgrade. The following sites/services may have experienced degraded performance or partial/full outages: Firefox Sync Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com) input.mozilla.com pulse.mozilla.org firefoxlive.mozilla.org demos.mozilla.org www.mozillademos.org www.drumbeat.org Background: There were two separate issues that we encountered,… Read more

Tags: ,

Categories: Outages

Mozilla Network Outage Report – 01/14/2011, 7:30pm PST – 11:30pm PST

Posted by mrz

At around 7:30pm PST Friday night Mozilla’s primary data center in San Jose went offline, affecting multiple services. CoreSite, Mozilla’s San Jose data center provider, indicated that they had lost city water and had suffered site-wide CRAC unit failures. These units provide cooling for the data center and without them, the ambient air temperature quickly… Read more

Tags: , ,

Categories: Outages

Mozilla Outage Report – mozilla.org DNSSEC – 09/16/2010

Posted by mrz

For several hours this morning, mozilla.org failed DNS resolution for sites that required DNSSEC validation. This appears to only have affected early DNSSEC adopters and not the larger widespread Internet. Background: As part of Thursday (September 16th) night’s scheduled maintenance, we had planned to upgrade Mozilla’s nameservers and enable DNSSEC for mozilla.org. Despite weeks of… Read more

Categories: Outages

Mozilla Network Outage Report – 11/12/2009, 5:07am PST – 5:20am PST

Posted by mrz

At around 5:07am PST this morning, Mozilla’s primary datacenter suffered an approximately 13 minute outage. Mozilla uses redundant Cisco Firewall Service Modules in the core Cisco 6509 switches. At about 5:07am the primary unit failed and appears to have crashed within the failover code. This caused an incomplete failover and the standby FWSM never assumed… Read more

Categories: Outages