The old store used to redirect non-North American users to the International Store, but unfortunately that redirect wasn’t carried over when we took the store offline.

I was inspired Sunday night to put together a ZXTM TrafficScript rule based off the code sample on Zeus’s knowledgehub (and because of bug 521914).

And then on Wednesday Zeus finally released ZXTM 6.0 with geolocation support!

What was 40 some lines of code is now like 5.

Before:

$ipaddr = request.getRemoteIP();

# Integer representation of $ipaddr >> 1
string.regexmatch( $ipaddr, "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+)" );
$ip = ((($1*256+$2)*256+$3)*128+$4/2);

$arr = resource.get( "geoip.dat" );

# initialize indices
$i = 0; $j = string.len( $arr )/6-1;

# $arr[$i] <= $ip < $arr[$j]
# iteratively halve the distance between $i and $j until they are adjacent

while( $j-$i > 1 ) {
   # midpoint between $i and $j
   $k = ($i+$j)/2;

   # compare $ip with $arr[$k]
   if( string.bytesToInt( string.subString( $arr, $k*6, $k*6+3 ) ) > $ip ) {
      $j = $k;
   } else {
      $i = $k;
   }
}

# Now, $arr[$i] <= $ip < $arr[$j] and $j == $i+1
# Look up the 2-character country code (returns '??' if unknown)
$ccode = string.subString( $arr, $i*6+4, $i*6+5 );

if ( ($ccode != "US") && ($ccode != "CA") ) {
}
else {
   http.redirect("https://intlstore.mozilla.org/");
}

After:

$ipaddr = request.getRemoteIP();

$country = geo.getCountryCode( $ipaddr );

if ( ($country != "US") && ($country != "CA") ) {
      http.redirect("https://intlstore.mozilla.org/");
}
​

I have a confession.  We secretly did something last night (we only barely announced it to Metrics).

No, we didn’t secretly replace the fine coffee at some four-star restaurant but pretty close.

Hot off our 2 second gain in average page load times for addons.mozilla.org, we shaved another 2 seconds off by duplicating The Amsterdam Reboot platform in Singapore (as a proof-of-concept).  Don’t take my word for it – take Gomez’s word:

A little background

For what seems like ages I’ve been trying to figure out how to best serve Asia-Pacific users. It’s a tough case to make because I didn’t have a method to easily measure how much bandwidth traffic I’d need or how it would change page load times or user perceptions.

But I’m a network guy and I’ve had this feeling that we need something in this region. We certainly have a growing population in the area – 5 of the top 9 countries we’ve seen > 20% growth in the last five months have been in Asia. I’ve only been lacking operational data.

Over the summer I’ve been playing and testing Voxel’s Silverlining Technology Preview, their global cloud computer platform.  On October 1, it’s going to move out of Preview and their cloud platform would be generally available in all their POPs, including Singapore.

Seemed like a good way to get data…

In a matter of hours I had spun up two Zeus ZXTM and one GLB cloud servers with Voxel in Singapore. I waited till our normal Thursday night window before turning it live.

A couple take away notes on this

1. We did it right.  We built a proxy/caching platform as part of The Amsterdam Reboot that can easily be replicated anywhere and instantly provide real quantifiable performance benefits.
2. Clouds make perfect sense to do proof-of-concepts.
3. IT can move really fast.  We had these three servers ready to go Wednesday morning in a couple hours..
4. Mozilla’s webdev crew does an awesome job writing extremely cachable webapps. I’m seeing a 91% cache hit rate (350,000 objects, 1.5GB).
5. If this is a sustainable location, pushing user-focused sites like support.mozilla.com to Singapore are next on my list.
6. I’d love to run this concept in other geographies like South America. Who does clouds down there?

What I most like about this platform is that it’ll allow us to strategically get content closer to users where it most makes sense.

This is right now just a proof-of-concept. It lets me experiment and get real metrics. I’m very interested in hearing from people who actually live in the area – does this make you happier?

One more thing

I was lucky to have two providers who stepped in and provided resources to let me run this POC. Both deserve a special thanks.

1. Zeus. These guys are great and quickly issued temporary licenses for both ZXTM and GLB.
2. Voxel. You guys have a nice platform in the right part of the world. I like working with you.

For a number of months we’ve been looking at Zeus and their ZXTM product.  It has some advantages (and some disadvantages) and one of the biggest is that it’s all software and I can quickly deploy it on any available hardware we have.  Zeus has been extremely helpful in quickly issuing unlimited node eval license keys on short notice (so thanks Jasper/Chris) and providing the level of supported I’d more likely expect to be given to a paying customer.

We’ve shifted two of the highest traffic back-end infrastructure services for Firefox over to a Zeus ZXTM cluster:

1. fxfeeds.mozilla.org – Live Bookmarks feed URL in a default Firefox install.
2. versioncheck.addons.mozilla.org – URL Firefox uses to check for Add-On updates.

fxfeeds.mozilla.org is nothing more than an RSS feed.  The site itself has no real content – there isn’t even a DocumentRoot defined in Apache.  In fact, the config is nothing but 40 HTTP redirects:

[root@mradm02 domains]# egrep 'Redirect|Rewrite' fxfeeds.mozilla.org.conf  | wc -l
40

versioncheck.addons.mozilla.org is my one Firefox 3 claim-to-fame.  Without that change, a lot of the scalability work we’re doing would be extremely difficult and require a lot more QA time.

Since Firefox checks for Add-On updates after updating itself (and periodically on its own), versioncheck.addons.mozilla.org is one of the sites that sees a large spike in traffic around release time.  It’s also highly cachable content and easy to scale with CPU (for SSL) and memory (for cache).

I moved fxfeeds.mozilla.org over yesterday and was astonished to see what sort of traffic that site alone generated – staggering might be a better way to say it.  Nearly 40Mbps of basically 302 redirects and upwards of 400,000 connections/second (and incidentally, moving that site off the Netscalers significantly dropped its resources issues).

versioncheck.addons.mozilla.org was moved Monday morning onto a test ZXTM cluster and again later this afternoon on a more production capable ZXTM cluster.  It’ll push 30Mbps on its own during non-release periods and 2-3x that during a full release.

Since no post is good without pictures, I grabbed two graphs from the ZXTM cluster.  The first shows connections/second and the other shows bandwidth (bits per second).

All I want for Christmas is a load balancer that does not suck.  It should do the following:

1. Terminate and offload SSL sessions at a rate of no less than 20,000/sec
2. Provide in-memory content caching
3. Support web logging, Apache-style (and despite what some hardware vendors thing, web logging is really important)
4. Have some method of scaling capacity (particularly at the SSL layer)

I know this is a lot to ask for but I’ve been good this past year.

We’re finding all sorts of uses for this now and have moved several websites over to this and have a couple more websites/services (like IRC) scheduled for this.

Purely out of interest I graphed the number of queries per hour through geodns from the two nameservers (one in San Jose, another in Amsterdam) and got:

I was concerned that two VMs wouldn’t work and I’d be looking at needing physical hardware but that hasn’t turned out to be the case yet (barely pushes more than 100Kbps with very little CPU load).  Although, since I’ve recently become a VMware DRS fan (like this guy) we’ve already cloned the San Jose nameserver and will be adding that as a third nameserver.  Since adding another nameserver/VM splits the load I also think this will scale nicely.

releases.mozilla.org is handled through round-robin DNS.  Regardless where you are in the world, you’ll get a “random” release mirror that may or may not be anywhere close to you and because of various physical reasons might induce a lot of latency and may or may not take a long time to download (admittedly, non-interactive transfers are less worried about latency but sitting in New York and downloading from China is going to take a long time).

As a network guy, I like to get my content as close to users as possible.  I know from my two visits to China that getting Add-ons updates from European mirror blow especially when I’m mere miles from our China release mirror!

After looking at a couple commercial solutions we decided to build our own geodns solution using the latest version of BIND and two patches to add Maxmind’s GeoIP functionality and to replace the backend zonefiles with MySQL.  That last patch lets us do all sorts of interesting thinks like nailing off bug 406267 (easily anyways, without having to munge text files).

Mark’s been working on all of the CLI management end to enable us to add or disable mirrors.  Hopefully I can get him to blog about the technical details but I’ll just add that prior to this I didn’t know anything about SQL views.

So when are we going deploy this?  Glad you asked.

On Tuesday, May 27 we’ll flip the switch and change releases.mozilla.org to a CNAME to releases.geo.mozilla.com, which, until Tuesday, will be in various states of testing but should work should be so inclined to try.

Currently, of the 15 release mirrors, 9 are in the US and 2 are in China.  The initial rollout will only directly benefit our Chinese users by directing them to the two Mozilla Online release mirrors in China.  Later we’ll try to do something on a geo-regional basis (but first, we’d want more than 3 mirrors in EU- anyone?)

This is just the tip of the iceberg.  Globally we use Netscaler-based GSLB to load balance and direct users to the network-wise closest site.  However, not all Mozilla properties are behind the Netscaler (or need to be) and we haven’t had a good way to serve users from different data centers unless they were.

Down the road we planning on adding some sort of weighting (not all mirrors or sites are equal), query statistics (where are lookups coming from?) and some sort of Nagios integration to automatically disable/enable a record.

And hopefully users will rejoice!

To check this I ran two weeks’ worth of web server logs through awstats with the geoip plugin.  From this, I’ve learned the following:

1. I was wrong.  In this post I said that “connectivity to mainland China can often be congested and can induce a lot of latency.”  Round-trip times between San Jose and China are only 30ms more than they are to Amsterdam (195ms vs. 166ms).  I don’t see any of the congestion that I saw from either of the two hotels I stayed or at that I see from the China office.
2. We’re serving a large portion of the Asia-Pacific region out of our Beijing, China colo. Far more than I think any of us had thought and we’re doing it remarkably well (even Gomez says so)!  I’ve even received emails from folks in that region wondering why www.mozilla.com is so much quicker/closer than any of Mozilla’s other web properties.

Looking at those log hits through awstats and pushing those results into Numbers gets me this:

(The 5% USA appears to be a combination of errors in the allocation database (IP is registered to a US entity but traceroute shows otherwise) and a few US-based ISPs that, for whatever reason, keep hitting China.  Other is a lot of smaller hits to Asia-Pacific countries and some IP addresses that didn’t automatically resolve to a country.)

Since the color seperation might be hard to read, the raw numbers are here.

I’m generally very happy with these results. In the next few weeks we should be adding addons.mozilla.org and spinning up our geo-dns setup to start pushing releases.mozilla.org out of China too.  Stay tuned.

In December I was back in China to install and deploy that data center and promised that “in the next several weeks we’ll be pulling up Mozilla websites in China.”

This past week[1] we did two important things for our Chinese users -

1. Integrated www.mozillaonline.com into our svn repo and pushed out its content to all static web clusters in Amsterdam, San Jose and China (it used to be hosted off a fileserver in the China office!). This site’s currently served out of San Jose and China.
2. Setup a mirror of releases.mozilla.org in China. This is currently known as releases.mozillaonline.com (until we get a more geo-aware DNS solution in place for releases.mozilla.org you’ll have to use that address).

This week we’ll make another important change.

During Tuesday night’s maintenance window (02/12 @ 8pm Pacific) I’ll be adding the China datacenter into www.mozilla.com‘s global load balancing (which I talked about here). As I mentioned in my last post, we’ll also be watching the server logs to make sure GSLB isn’t sending users outside of China to China when they’d be better served out of San Jose or Amsterdam. If you find yourself in this category, please let me know (email or file a bug).

All of this effort has been to get content as close as possible to users. From my two trips to China I know getting to Mozilla content is often painfully slow, reminiscent of my pre-ISDN days. Updating Minefield nightlies every morning was often a couple hour process!

[1] I started this post with the intention of posting it January 28th but got side tracked when the host that holds all web content (before pushing out to the web servers) and all of the releases.mozilla.org mirror content lost it’s storage shelf, halting all of these plans. Because of this storm it took longer than usual to get resolved.

The algorithm first checks to see if there’s a match in the static maps and then falls back to proximity metrics. If that’s missing, it’ll round-robin through all the GSLB sites (effectively three – San Jose, Amsterdam and China – though technically it’s six and actually it’ll only round-robin through sites that have that service/site defined).

In practice, this round-robin fallback hasn’t been terribly user impacting mostly. You’d only ever hit the round-robin fallback if no one in your network (/16 in this case, but based on the source address of whatever name server you used) had ever accessed any of Mozilla’s web properties. And while that’s possible, the performance his is really minimal. But then up until now, we’ve only had two GSLB sites to send users to and San Jose and Amsterdam are well connected.

China changes things. Connectivity to mainland China can often be congested and can induce a lot of latency (which matters less for downloads than it does for interactive sites). If you’re in New York and fall through to round-robin, you very likely don’t want to end up taking 80ms to the west coast and another 300ms-400ms to China. This is a bad user experience.

If you don’t believe me, look at my trip times from San Jose to some Mozilla gear in China:

mrz@boris [~/] 15> ping mozcn01
PING mozcn01 (10.86.35.203) 56(84) bytes of data.
64 bytes from mozcn01 (10.86.35.203): icmp_seq=6 ttl=237 time=389 ms
64 bytes from mozcn01 (10.86.35.203): icmp_seq=9 ttl=237 time=439 ms

mrz@boris [~/] 17> ping releases.mozillaonline.com
PING releases-cn.glb.mozilla.com (59.151.50.34) 56(84) bytes of data.
64 bytes from 59.151.50.34: icmp_seq=0 ttl=231 time=206 ms
64 bytes from 59.151.50.34: icmp_seq=1 ttl=231 time=212 ms
64 bytes from 59.151.50.34: icmp_seq=2 ttl=231 time=205 ms

Since December I’ve been working on ways to exclude China from the fallback round-robin method while making sure our Chinese users don’t goto San Jose or Amsterdam.  The latter was easier – using MaxMind’s GeoLite database I’ve built a static proximity location map assigning all IP addresses in China to the China datacenter.  The former, excluding China from round-robin, took longer to figure out.

One of the GSLB methods the Netscalers have is a “weighted round-robin” and while it’s not documented, it turns out that weights apply even to the fallback round-robin method.  Weights range from 1 to 100.  So now I can do something like:

bind gslb vserver glb-mozcom -serviceName mozcom-80-sj -weight 100
bind gslb vserver glb-mozcom -serviceName mozcom-80-cn01
bind gslb vserver glb-mozcom -serviceName mozcom-80-cn02
bind gslb vserver glb-mozcom -serviceName mozcom-80-nl01 -weight 50
bind gslb vserver glb-mozcom -serviceName mozcom-80-nl02 -weight 50

which will send 200 requests  to either San Jose or Amsterdam before sending -1- request to China.

It’ll be interesting to see what the actual user impact of this is and if anyone not in mainland China ends up hitting China.  We’ll be testing this for a week, starting next Tuesday, with www.mozilla.com and looking at the web server logs to see where users are coming from.

I’d also be interested in feedback from you if you find yourself in China and you should be elsewhere or your find yourself in China and www.mozilla.com is quicker for you!

[1]GSLB works by looking up the IP address of the host that made a DNS request to somehost.glb.mozilla.com and determining which data center is closest.   This is a fairly common method to do global load balancing.

This whole saga’s been detailed elsewhere (and here and here and here).

The Good (and the graphs)
Last time, I rolled back when Europe started waking up (~12:30am PDT). The Netscalers hit close to 900 SSL transactions/second before failing. As you can see, we more than doubled that rate!

addons.mozilla.org is highly cache-friendly, which makes this whole solution work really well. Don’t believe me? Check out how the slope of cache request/second match nicely with SSL transactions/second:

Where is everyone?
And finally, because everyone likes graphs and charts, here’s what AMO traffic looks like out of Amsterdam (and bonus points to anyone who can tell me when we made the DNS change):

… and out of San Jose:

John Lilly blogged about Firefox growth based on language (locale) version, and while the US probably accounts for the majority of the traffic out of San Jose, the US is not, as John points out, the majority. 55% of AMO’s traffic is coming out of Amsterdam vs. 44% out of San Jose.

ps. To give credit where credit’s due, I should thank Citrix for aborting their normal release process to get me new code builds to get around two potential service impacting bugs. Took them a mere few days vs. six weeks, so rock on Citrix.