1. Firefox (of course)
2. HTML5
3. Geolocation
4. Do Not Track

IT/Operations is often the unglamorous side of Mozilla. We work in the shadows and you generally don’t see us unless something’s broken. However, we want to push technology forward too. We like to be thought leaders. We like to solve the “chicken-and-egg” problem. For example, over the summer of 2010, we laid the infrastructure ground work for DNSSEC and published a signed mozilla.org zone.

This year we’re ready for another challenge and are excited to join others in support of the World IPv6 Day! June 8, 2011 will be the first global-scale “test flight” of IPv6 and we’re going to be a part of it. This will also be our chance to see which parts of the network already support IPv6 and which parts require software (or hardware) upgrades.

As part of World IPv6 Day, we’ll offer a couple marquee Mozilla web properties over IPv6. We’ll be able to measure how many users are already dual-stack (both IPv4 & IPv6) vs. those who don’t yet have IPv6 capabilities. In short, we’ll have a view on the state of the Internet’s IPv6 end-user deployment.

What is IPv6?
IPv6 is the next generation protocol for the Internet. The current protocol, IPv4, is based on 32-bit addressing and has the familiar syntax of 63.245.209.10. Unfortunately, after more than 30 years, the Internet is running out of available IPv4 address space.

IPv6 (and it’s 128-bit addressing) provides a vast increase in available address space – 4 billion times the number that are available under IPv4. Addresses take on an entirely new syntax – 2620:101:8003:200:217:f2ff:fe09:d8ea, for example.

When I started at Mozilla in 2006, I continued the IPv6 work I was doing at my last job and for some time now Mozilla’s had a production v6 network announced out of our San Jose data center. I even experimented with a couple websites back in 2007. Unfortunately, there’s been a lack of user demand and, as such, the “chicken-and-egg” problem.

We recently traded in our original /48 allocation in exchange for a /44 and are slowly renumbering and bringing up native IPv6 peering with Mozilla’s transit providers. My Network Engineers even pushed out native IPv6 connectivity within Mozilla’s Mountain View office.

Those interested in tracking our progress can follow along in bug 630581.

I hope you’ll follow along with us in the coming months and join us for World IPv6 Day on June 8, 2011!

The old store used to redirect non-North American users to the International Store, but unfortunately that redirect wasn’t carried over when we took the store offline.

I was inspired Sunday night to put together a ZXTM TrafficScript rule based off the code sample on Zeus’s knowledgehub (and because of bug 521914).

And then on Wednesday Zeus finally released ZXTM 6.0 with geolocation support!

What was 40 some lines of code is now like 5.

Before:

$ipaddr = request.getRemoteIP();

# Integer representation of $ipaddr >> 1
string.regexmatch( $ipaddr, "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+)" );
$ip = ((($1*256+$2)*256+$3)*128+$4/2);

$arr = resource.get( "geoip.dat" );

# initialize indices
$i = 0; $j = string.len( $arr )/6-1;

# $arr[$i] <= $ip < $arr[$j]
# iteratively halve the distance between $i and $j until they are adjacent

while( $j-$i > 1 ) {
   # midpoint between $i and $j
   $k = ($i+$j)/2;

   # compare $ip with $arr[$k]
   if( string.bytesToInt( string.subString( $arr, $k*6, $k*6+3 ) ) > $ip ) {
      $j = $k;
   } else {
      $i = $k;
   }
}

# Now, $arr[$i] <= $ip < $arr[$j] and $j == $i+1
# Look up the 2-character country code (returns '??' if unknown)
$ccode = string.subString( $arr, $i*6+4, $i*6+5 );

if ( ($ccode != "US") && ($ccode != "CA") ) {
}
else {
   http.redirect("https://intlstore.mozilla.org/");
}

After:

$ipaddr = request.getRemoteIP();

$country = geo.getCountryCode( $ipaddr );

if ( ($country != "US") && ($country != "CA") ) {
      http.redirect("https://intlstore.mozilla.org/");
}
​

I have a confession.  We secretly did something last night (we only barely announced it to Metrics).

No, we didn’t secretly replace the fine coffee at some four-star restaurant but pretty close.

Hot off our 2 second gain in average page load times for addons.mozilla.org, we shaved another 2 seconds off by duplicating The Amsterdam Reboot platform in Singapore (as a proof-of-concept).  Don’t take my word for it – take Gomez’s word:

A little background

For what seems like ages I’ve been trying to figure out how to best serve Asia-Pacific users. It’s a tough case to make because I didn’t have a method to easily measure how much bandwidth traffic I’d need or how it would change page load times or user perceptions.

But I’m a network guy and I’ve had this feeling that we need something in this region. We certainly have a growing population in the area – 5 of the top 9 countries we’ve seen > 20% growth in the last five months have been in Asia. I’ve only been lacking operational data.

Over the summer I’ve been playing and testing Voxel’s Silverlining Technology Preview, their global cloud computer platform.  On October 1, it’s going to move out of Preview and their cloud platform would be generally available in all their POPs, including Singapore.

Seemed like a good way to get data…

In a matter of hours I had spun up two Zeus ZXTM and one GLB cloud servers with Voxel in Singapore. I waited till our normal Thursday night window before turning it live.

A couple take away notes on this

1. We did it right.  We built a proxy/caching platform as part of The Amsterdam Reboot that can easily be replicated anywhere and instantly provide real quantifiable performance benefits.
2. Clouds make perfect sense to do proof-of-concepts.
3. IT can move really fast.  We had these three servers ready to go Wednesday morning in a couple hours..
4. Mozilla’s webdev crew does an awesome job writing extremely cachable webapps. I’m seeing a 91% cache hit rate (350,000 objects, 1.5GB).
5. If this is a sustainable location, pushing user-focused sites like support.mozilla.com to Singapore are next on my list.
6. I’d love to run this concept in other geographies like South America. Who does clouds down there?

What I most like about this platform is that it’ll allow us to strategically get content closer to users where it most makes sense.

This is right now just a proof-of-concept. It lets me experiment and get real metrics. I’m very interested in hearing from people who actually live in the area – does this make you happier?

One more thing

I was lucky to have two providers who stepped in and provided resources to let me run this POC. Both deserve a special thanks.

1. Zeus. These guys are great and quickly issued temporary licenses for both ZXTM and GLB.
2. Voxel. You guys have a nice platform in the right part of the world. I like working with you.

During a normal release we have tools we can use to adjust the rate at which we offer updates. We use this to reduce load on the back end systems or to help reduce load on the download mirrors.

Our preference is to do a release completely unthrottled so users get timely updates.

During the Firefox 3.0.6 release we had a number of system problems that prevented us from releasing updates unthrottled. These were all detailed in the Post Mortem.

To the Operations Team’s credit (and I’m serious here), most of those issues were removed prior to yesterday’s Firefox 3.0.7 release and by 9am this morning we were cranking along – no throttling.

Unfortunately the Mirror Network started showing pressure and instead of throttling back on the release, we opted to augment the Mirror Network with our own download servers in San Jose.

That pushed our aggregate bandwidth out of San Jose to nearly 3Gbps:

At around this time offsite monitors starting alerting about a sharp increase in page load times to various Mozilla website properties. Took a bit to track down but the newly turned up Level 3 peer was saturated:

Any outbound traffic whose best route was out through Level3 was impacted. We fixed this temporarily by turning down Level3.

(I should note that our design requirements for upstream transit is at least two connections per provider so we can push 2Gbps. Level 3 is no exception, however, the second connection has been offline because Derek was seeing a lot of packet loss across the optical connection which coincidentally got resolved today.)

These problems are solvable and we’ve had plans to put tools in place to balance load during situations like this. Unfortunately, today’s issues came up a lot quicker than we had planned.

A couple things we’ll be looking at before the next release:

1. Evaluating Internap’s FCP to dynamically shift traffic based on cost and performance metrics. (And as luck would have it, this showed up this afternoon!)
2. Looking to see how we can better balance outbound traffic outside of using FCP.
3. Adding capacity to our Mirror Network (can you help?).
4. Evaluating options around upgrading from several 1GE upstream connections to 10GE connections.

This is a great problem to have, to be sure, and a far cry from the panic three years ago of “OMG we’re about to push 100Mbps!”.

I’m really interested in how others have gone about solving problems like this. Leave me comments.

I’ve been working on a couple capacity planning projects and have been knee deep in bandwidth metrics. That, combined with turning up Level3 yesterday got me looking more into where that bandwidth is coming from (and Reed asked).

The first chart shows a breakdown by protocol. Shouldn’t be any surprise that 82% of Mozilla’s traffic is web related (SSL being the larger which, of course, make sense since it is out to destroy me). The “Other” category was filled with services under 1%.

I took a look at the same bandwidth data and broke it down by source.  Most of the traffic is from addons.mozilla.org (which isn’t surprising – it alone causes all my SSL scaling headaches).

The other sites aren’t too surprising to me after a couple rounds of load balancer testing.  I expected and do see fxfeeds.mozilla.com (Firefox Live Bookmarks), versioncheck.addons.mozilla.org and services.addons.mozilla.org.

We’ve grown (bandwidth-wise) to the extent that having two transit providers wasn’t sufficient. Should either fail, I wouldn’t have felt comfortable pushing ~800Mbps out a single provider without any backup.

The trick will be making sure to hit bandwidth commits across all three…

We ran into several issues and postponed the upgrade on core2 until those issues are resolved.

1. The switch’s Compact Flash cards weren’t formatted in a format that rommon (the low level boot loader) could read and the switch failed to load any OS when rebooted (bug 473084).
   

   <rant>
   Unfortunately IOS didn’t flag that as an error when reading/writing to it. It didn’t even flag an error when the boot variable was set to boot off of it which is stupid because IOS clearly knew as shown in the log when I had remote-hands re-seat the card:

   Jan 11 22:44:23 core2 3927937: Jan 11 22:44:21.978 PDT: %PCMCIAFS-SP-5-DIBERR: PCMCIA disk 0 is formatted from a different router or PC. A format in this router is required before an image can be booted from this device
   </rant>

2. Two of the VMware ESX storage arrays are single-homed, connected to one switch (bug 473113).  This is fallout from previous NetApp performance issues that were forgotten and never addressed and caused a number of build VMs to go offline (bug 473112) .
3. A number of non-user facing, multi-homed hosts went offline.  All of the RHEL Linux servers have an active/standby network setup.  In several cases the standby interface didn’t work or wasn’t properly configured.  This was more of an annoyance to the IT Team than to anyone else but did cause outages for some backend services (most notably the VMware VC server and mradm01, one of the Nagios servers).

We’ll be addressing those issues before scheduling the remaining upgrade to core2. We’ll also be looking at implementing some routine (perhaps quarterly) test of the infrastructure in a controlled environment to ensure its high “availbility-ness”.

Tonight’s one of those rare exceptions. Derek is upgrading both core1 and core2 to pickup software support for Cisco’s ACE module. We’re planning on using this in conjunction with the Zeus ZXTM cluster we setup.

A little sad though.  It’s going to reset uptime:

core1#sh ver | inc uptime
core1 uptime is 2 years, 28 weeks, 2 days, 20 hours, 40 minutes

core2#sh ver | inc uptime
core2 uptime is 2 years, 28 weeks, 2 days, 20 hours, 51 minutes

Love ‘em or hate ‘em, Cisco gear is solid.

For a number of months we’ve been looking at Zeus and their ZXTM product.  It has some advantages (and some disadvantages) and one of the biggest is that it’s all software and I can quickly deploy it on any available hardware we have.  Zeus has been extremely helpful in quickly issuing unlimited node eval license keys on short notice (so thanks Jasper/Chris) and providing the level of supported I’d more likely expect to be given to a paying customer.

We’ve shifted two of the highest traffic back-end infrastructure services for Firefox over to a Zeus ZXTM cluster:

1. fxfeeds.mozilla.org – Live Bookmarks feed URL in a default Firefox install.
2. versioncheck.addons.mozilla.org – URL Firefox uses to check for Add-On updates.

fxfeeds.mozilla.org is nothing more than an RSS feed.  The site itself has no real content – there isn’t even a DocumentRoot defined in Apache.  In fact, the config is nothing but 40 HTTP redirects:

[root@mradm02 domains]# egrep 'Redirect|Rewrite' fxfeeds.mozilla.org.conf  | wc -l
40

versioncheck.addons.mozilla.org is my one Firefox 3 claim-to-fame.  Without that change, a lot of the scalability work we’re doing would be extremely difficult and require a lot more QA time.

Since Firefox checks for Add-On updates after updating itself (and periodically on its own), versioncheck.addons.mozilla.org is one of the sites that sees a large spike in traffic around release time.  It’s also highly cachable content and easy to scale with CPU (for SSL) and memory (for cache).

I moved fxfeeds.mozilla.org over yesterday and was astonished to see what sort of traffic that site alone generated – staggering might be a better way to say it.  Nearly 40Mbps of basically 302 redirects and upwards of 400,000 connections/second (and incidentally, moving that site off the Netscalers significantly dropped its resources issues).

versioncheck.addons.mozilla.org was moved Monday morning onto a test ZXTM cluster and again later this afternoon on a more production capable ZXTM cluster.  It’ll push 30Mbps on its own during non-release periods and 2-3x that during a full release.

Since no post is good without pictures, I grabbed two graphs from the ZXTM cluster.  The first shows connections/second and the other shows bandwidth (bits per second).