At around 1:30pm PDT Sunday afternoon Mozilla’s primary data center in San Jose went offline.

CoreSite, Mozilla’s San Jose data center provider, indicated that they had suffered a CRAC unit failure on the 16th floor. This unit provided the majority of the cooling for Mozilla’s cage. Without it, the ambient air temperature quickly rose to about 120° F.

To prevent thermal damage, the servers automatically shut themselves down.

Mozilla IT was onsite by 2:50pm PDT. By 7pm CoreSite had brought the ambient air temperature under 90° F. Mozilla IT had the majority of the infrastructure back online by 8:30pm PDT and declared everything “all clear” by 9:30pm PDT.

We apologize for any inconvenience this may have caused. We are working with CoreSite to better understand the points of failure and how they will work to prevent a re-occurrence.