You do.

I can break down the machines in the data center into three sections (and this is a very simplistic view):

1. Classic infrastructure. All the web sites and infrastructure to support Mozilla and Firefox.
2. Release Engineering. All the infrastructure that builds and tests Firefox.
3. Services. Right now mostly Firefox Sync but will soon include BrowserID & the App Store.

Each of these scale and grow at different rates but combined come out to about 12kW/month.

In laying out racks in a data center, you can either contain the cold air or contain the hot air (or do nothing). In this post, Tim talks about both options and why we’re using cold aisle containment.

What is “containment” and why is it important in a data center?

The containment of cold air supply or hot air return in data centers is a practice used primarily to maximize the efficiency of the cooling supply and heat rejection systems.  This helps minimize the cost of running and maintaining these systems. As this cost is being passed on to Mozilla, minimizing it as much as practicable is an important goal of this project.

Containment is also a means of ensuring that the cool air required by today’s high-density server installations is supplied in the correct volumes and at the correct temperature for these servers to operate as efficiently as possible. Since power is expensive, ensuring optimal server efficiency is also a very important goal of this project.

Why “cold aisle” containment?

The data center provider Mozilla chose, Vantage Data Centers, engineered the data center floor to deliver cold air from below and return hot air from above to the central cooling plant. Further, they provided the perforated tiles required to support a cold aisle installation. This saved Mozilla a good deal of money but, equally as important, it is the right solution for the power density Mozilla intends to have in each of its cabinets (i.e. up to 12 kW per cabinet).

Courtesy http://www.42u.com/cooling/cold-aisle-containment.htm

In high power density server installations (e.g. blade servers) , it is imperative that cool air be delivered at the right volume and temperature to the gear. Cold aisle containment ensures that:

• Hot and cold air aren’t mixed
• Cold air temperature can be monitored and controlled
• Cold air supply volumes can be monitored and controlled

As Mozilla’s needs grow and change over time, we can monitor and control the environment to meet the needs of not only today’s server and network infrastructure but tomorrow’s as well.

What about “hot aisle” containment?

Hot aisle containment would work best in a data center environment completely dedicated to Mozilla. In other words, containing hot air implies that the cold air supplied to the data center floor must be somehow “contained” by the boundaries of the data center itself (i.e. ceilings, walls, doors, floors, etc).

Courtesy http://www.42u.com/cooling/hot-aisle-containment.htm

In a data center environment shared with other tenants, as is the case with Mozilla’s space at Vantage (we’re sharing with two other tenants), Mozilla would not be able to monitor or control the temperature or volume of the cold air being supplied to the servers.

Further, Mozilla has no way of knowing (or influencing) the efficiency of other tenant’s designs and installations so the ambient air temperature for the data center module serving all three tenants (we’re only separated by cage walls) is very likely to fluctuate and be higher than the temperature ranges recommended by ASHRAE [edit: ASHRAE helps define recommended data center temperature limits].

ps. Special thanks to http://www.42u.com/ for their hot & cold aisle graphics.

Quick Introduction

We’re building out a new 1MW data center in Santa Clara, CA at Vantage Data Centers!

Project: SCL3 is our latest data center build out. Ever since we started talking about this internally we knew we wanted to blog about it, to talk about how we’re doing it, why we’re doing it, and why we’re doing it this way or that way.

Be open about it.

This is just the first of many blog posts. Through this blog we’ll talk about this process. We’ll share with you what drove us here, why we picked the rack layout we did, why we provisioned the power we did.

We’ll look towards you, too, to give input on some of the directions we’ve taken. Some of you are better experts than we are.

What is “SCL3″?

We use a three letter code followed by a number to name our facilities, both Mozilla Spaces and data centers.  This code seems reasonably mnemonic and doesn’t collide with too many three letter airport/IATA codes.  You’ll see us use these codes when we talk about hardware or servers at various locations.

We have three data centers in the Bay Area and are building a fourth:

1. SJC1: 55 So. Market Street, Market Post Tower (MPT)
2. SCL1: Internap, Santa Clara
3. SCL2: Layer42, Santa Clara
4. SCL3: Vantage, Santa Clara

We have others like PHX1, AMS1.

Quick Background

Back in July 2006 Mozilla moved from a small collection of ten racks to its first data center at 55 So. Market Street, San Jose.

Within the next year Mozilla IT opened a small presence in Amsterdam and followed it up with a presence in Beijing (which, coincidentally, we’re expanding in Q4). In 2010 Mozilla IT opened up a location in Phoenix at i/o Data Centers.

Towards the end of 2009 we knew were going to eventually run out of provisioned power in San Jose. Throughout 2010 and into 2011 Mozilla IT picked up additional data center space in the Bay Area with Layer42 & Internap to accommodate our continued growth.

In the Bay Area alone we’re consuming about 500kW. When we shopped around for what eventually became the Phoenix location, we knew that 500kW was around the tipping point to transition from retail to wholesale data center space.

As we started 2011, we started to think about “what do we do next?”. What do we need so we can move quickly, so we can handle the next generation of problems?

We knew:

• We need power. Data centers are really about providing power (and cooling it). We’ll run out of power before space.
• Operationally, managing three locations in the same area is harder. We want to consolidate.
• The infrastructure, all the web servers that support Mozilla, keeps growing.
• We hate downtime

Mozilla IT spent most of Q1/Q2 2011 searching around the Bay Area.  Sometime over the summer we decided to partner with Vantage Data Centers and have been extremely busy since then in lease negotiations and in getting started on the build-out of the space.

Over the next several weeks, we’ll talk about some of the designs, some of the reasons we needed more space/power and show pictures & video of the space.

ps. You’ve already joined the Mozilla Community Directory @ https://mozillians.org/ right?  Read this blog post if you forgot why it’s important!

• 10:00pm PDT (0500 UTC) DNSSEC Deployment. Mozilla will start publishing the following signed zones:
  • mozilla.com (bug 602424)
  • mozilla.net (bug 602423)

  This is a continuation of the work we started last summer with mozilla.org. We will monitor this for 24 hours before submitting the respective DS keys to our registrar. No downtime expected.

The following sites/services may have experienced degraded performance or partial/full outages:

• Firefox Sync
• Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com)
• input.mozilla.com
• pulse.mozilla.org
• firefoxlive.mozilla.org
• demos.mozilla.org
• www.mozillademos.org
• www.drumbeat.org

Background:
There were two separate issues that we encountered, both tracked in bug 639745.

1. DHCP relay failures. This is a regression in the JunOS code.

   Just before 10:00pm Monday night, multiple hosts in Phoenix began to lose their DHCP leases and drop offline. For reasons not yet understood, the DHCP relay feature was no longer operational.

   This caused an 8 minute outage for support.mozilla.com.

2. High CPU load. We began experiencing high (maximum) CPU usage on multiple FPCs after upgrading from 10.1 to 10.4R2. This did not have any immediate impact and we opted to continue working with overnight with JTAC on resolution.

   This morning as general load increased, this became a service impacting issue. Netops downgraded to 10.3R2.11 and eventually downgraded to 10.2S7 to resolve these issues.

We apologize for any inconvenience this may have caused and will continue to work with Juniper to understand why this failed and on a long term remedy.

• 6:00pm PST (0100 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The nature of this upgrade requires redundancy to be disabled and as such, all sites will experience a 10-15 second outage. While the actual user-facing outage is less than a minute, the entire window will be 2 hours.

  The following services will be impacted during this upgrade:
  

  • Firefox Sync
  • Socorro (crash-stats.mozilla.com & crash-reports.mozilla.com)
  • input.mozilla.com
  • pulse.mozilla.org
  • firefoxlive.mozilla.org
  • demos.mozilla.org
  • www.mozillademos.org
  • www.drumbeat.org

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 4:00pm PST (0000 UTC) support.mozilla.com primary data center migration. In preparation for Firefox 4, we’ll be moving the primary data center serving support.mozilla.com to Phoenix (bug 622752). During the migration the site will remain online but in a read-only mode. Duration 1 hour.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 3:00pm PST (2300 UTC) getpersonas.com update. We’ll be updating getpersonas.com to pick up code updates (bug 630990). The site will remain up and online during this push, however, user logins will be disabled for the duration of the database update. Duration 2 hours.
• 7:00pm PST (0300 UTC) CVS mirror firmware upgrade. We’ll be applying vendor recommended firmware updates to the servers behind cvs-mirror.mozilla.org which will require a reboot. (Last week’s attempt was aborted.) Duration 15 minutes.
• 9:00pm PST (0500 UTC) Phoenix Firewall upgrades. We’ll be picking up vendor-recommended software upgrades. The firewalls operate in redundant mode, and will be failed over after insuring the upgraded one is functioning properly. No downtime expected.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.

• 7:00pm PST (0300 UTC) getpersonas.com storage migration. We’ll be moving storage around in preparation for upcoming work. Unfortunately this requires taking parts of the site offline while data is moved. Duration 1 hours.
• 7:00pm PST (0300 UTC) CVS mirror firmware upgrade. We’ll be applying vendor recommended firmware updates to the servers behind cvs-mirror.mozilla.org which will require a reboot. Duration 15 minutes.

Please let me know if you have any reason why we should not proceed with this planned maintenance. As always, we aim to keep downtime to as little as possible, but unexpected complications can arise causing longer downtime periods than expected. All systems should be operational by the end of the maintenance window.

Feel free to comment directly if you see issues past the planned downtime.