The old store used to redirect non-North American users to the International Store, but unfortunately that redirect wasn’t carried over when we took the store offline.

I was inspired Sunday night to put together a ZXTM TrafficScript rule based off the code sample on Zeus’s knowledgehub (and because of bug 521914).

And then on Wednesday Zeus finally released ZXTM 6.0 with geolocation support!

What was 40 some lines of code is now like 5.

Before:

$ipaddr = request.getRemoteIP();

# Integer representation of $ipaddr >> 1
string.regexmatch( $ipaddr, "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+)" );
$ip = ((($1*256+$2)*256+$3)*128+$4/2);

$arr = resource.get( "geoip.dat" );

# initialize indices
$i = 0; $j = string.len( $arr )/6-1;

# $arr[$i] <= $ip < $arr[$j]
# iteratively halve the distance between $i and $j until they are adjacent

while( $j-$i > 1 ) {
   # midpoint between $i and $j
   $k = ($i+$j)/2;

   # compare $ip with $arr[$k]
   if( string.bytesToInt( string.subString( $arr, $k*6, $k*6+3 ) ) > $ip ) {
      $j = $k;
   } else {
      $i = $k;
   }
}

# Now, $arr[$i] <= $ip < $arr[$j] and $j == $i+1
# Look up the 2-character country code (returns '??' if unknown)
$ccode = string.subString( $arr, $i*6+4, $i*6+5 );

if ( ($ccode != "US") && ($ccode != "CA") ) {
}
else {
   http.redirect("https://intlstore.mozilla.org/");
}

After:

$ipaddr = request.getRemoteIP();

$country = geo.getCountryCode( $ipaddr );

if ( ($country != "US") && ($country != "CA") ) {
      http.redirect("https://intlstore.mozilla.org/");
}
​

That’s the amount of time we shaved off average page load times for addons.mozilla.org after last night’s work. The Amsterdam Reboot in effect!

It’s been nearly ten months since we served production traffic at these levels out of Amsterdam.

I remember coming back and was up late at night (fighting jetlag) setting up the Netscaler load balancers. By early January we had a CVS mirror up and running and a week later had staged www.mozilla.com and www.mozilla.org in Amsterdam. By the middle of February we had shifted European production traffic over to Amsterdam.

By May of 2007 (oh and here) we started serving addons.mozilla.org out of Amsterdam too.

Since then, the San Jose data center has grown from seven racks to twenty-four and nearly 500 servers (for the sake of this post I’m counting the 150 Mac Minis as “servers”).

Unfortunately, Amsterdam hasn’t seen the same sort of server growth and in the past half year or so we’ve had to pull sites back from Amsterdam and serve them from San Jose only. When the load balancers there could no longer handle the SSL traffic in January we stopped serving addons.mozilla.org out of Amsterdam too.

The Reboot

On September 2, we’ll reboot Amsterdam. Much like The Six Million Dollar Man, “… we can rebuild [it]. We have the technology… We can make [it] better than [it] was before. Better, stronger, faster.”

This morning we shipped a half loaded HP c7000 BladeSystems chassis out to Amsterdam. Next week both Arzhel and Derek will be in Amsterdam to deploy new servers and turn down some of the old legacy hardware.

One of the issues with the current Amsterdam deployment is that we can only really serve static sites – sites that don’t have a database behind them. It’s a bit more complicated to replicate databases to remote locations and keep them in sync with San Jose.

Based on our success with the Zeus ZXTM platform, we’re deploying a Nehalem based ZXTM cluster. Much like we originally did with addons.mozilla.org, we’ll have a platform where we can proxy/cache any Mozilla website and serve it out of Amsterdam.

We’ll also be deploying a new global load balancing system that Arzhel’s spent time staging and getting ready for production.

The Plan

During tomorrow night’s downtime window we’ll start the process of temporarily shutting down Amsterdam. We’ll pull back all websites to San Jose and stop serving web content out of Amsterdam for about two weeks.

We’re doing this in advance of the actual deployment to make sure we don’t hit any surprises.

What won’t be affected?

There are two VMware ESX servers in Amsterdam that we won’t be touching. A quick list of hosts that won’t be affected by this are:

• dn-vcs01
• gravel
• geodns02
• l10n-01
• rhino01
• sea-qm-centos5-01
• sea-qm-win2k3-01
• konigsberg

I’m excited about this. I’ve said before I’m still a network guy at heart and I really like getting content closer to consumers. I’m hoping this Reboot becomes an easily deployable platform for other parts of the globe.

After a couple hours this week re-learning Perl and learning SOAP I have something that I think is cool - @mozdashboard!

I experimented a bit earlier in this week before Zandr gave me an idea of what I was really looking for with this.  This version keeps state using Config::IniFiles (yeah, I said it and it was surely an easy way out) and tweets when it detects new highs.

zxtmtwitter.pl tracks the following sites:

1. addons.mozilla.org
2. versioncheck.addons.mozilla.org
3. fxfeeds.mozilla.com

It’ll tweet if it detects a new bandwidth high (either inbound or outbound) or a new simultaneous current connection count.

The program’s not without some faults – I’m not accounting for any counter wraps and I’m not happy with some of the hackery to poll each ZXTM node (I can’t find a way to poll one and get the cluster aggregates).

Code and .ini below the fold.

Config File (zxtm.ini):

[addons.mozilla.org]
vserver=addons-ssl
maxconnections=37327
bytesOut=39293647822458
bytesIn=4266246130636
timestamp=1246168046
maxbpsOut=82
maxbpsIn=8

[versioncheck.amo]
vserver=versioncheck.addons.mozilla.org
maxconnections=18556
bytesOut=6227643301317
bytesIn=4356691185954
timestamp=1246168043
maxbpsOut=11
maxbpsIn=8

[fxfeeds.mozilla.com]
vserver=fxfeeds
maxconnections=145780
bytesOut=7943190385192
bytesIn=6738548312076
timestamp=1246168041
maxbpsOut=17
maxbpsIn=14

Perl code (zxtmtwitter.pl):

#!/usr/bin/perl

use warnings;
use strict;
use Config::IniFiles;
use SOAP::Lite 0.60;

my $DEBUG = 0;

## twitter
     my $tweetusr="mozdashboard";
     my $tweetpwd="myPassword";
     my $updateURL="http://twitter.com/statuses/update.xml";
##

## ZXTM
     my $admin_server = 'https://myUser:myPassword';

my $conn1 = SOAP::Lite -> uri('http://soap.zeus.com/zxtm/1.0/System/Stats/') -> proxy("$admin_server\@zlb01:9090/soap");
my $conn2 = SOAP::Lite -> uri('http://soap.zeus.com/zxtm/1.0/System/Stats/') -> proxy("$admin_server\@zlb02:9090/soap");
my $conn3 = SOAP::Lite -> uri('http://soap.zeus.com/zxtm/1.0/System/Stats/') -> proxy("$admin_server\@zlb03:9090/soap");
my $conn4 = SOAP::Lite -> uri('http://soap.zeus.com/zxtm/1.0/System/Stats/') -> proxy("$admin_server\@zlb04:9090/soap");
my $conn5 = SOAP::Lite -> uri('http://soap.zeus.com/zxtm/1.0/System/Stats/') -> proxy("$admin_server\@zlb05:9090/soap");

my %ini;
tie %ini, 'Config::IniFiles', ( -file => "zxtm.ini" );

# loop through each section
foreach( keys %ini ) {
     my @vserverArray;
     my $timeStamp = time();

     print "Found section [ $_ ], " if $DEBUG;

     my $vserver = $ini{$_}{"vserver"};
     print "vserver == $vserver\n" if $DEBUG;

     my $old_Max = $ini{$_}{"maxconnections"};
     my $old_BytesOut = $ini{$_}{"bytesOut"};
     my $old_BytesIn = $ini{$_}{"bytesIn"};
     my $old_MaxBpsOut = $ini{$_}{"maxbpsOut"};
     my $old_MaxBpsIn = $ini{$_}{"maxbpsIn"};
     my $old_Timestamp = $ini{$_}{"timestamp"};

     undef @vserverArray;
     push @vserverArray, $vserver;

     my $max = &getMaxConn([@vserverArray]);
     my $BytesOut = &getBytesOut([@vserverArray]);
     my $BytesIn = &getBytesIn([@vserverArray]);

     print "\tmaxConn = $max ($old_Max)\n " if $DEBUG;

     my $timeDiff = $timeStamp - $old_Timestamp;

     # The number eight keeps rendering as a smiley face.
     # I replaced eight with (4*2) so it doesn't look stupid.
     my $bpsOut = int((($BytesOut - $old_BytesOut) * (4*2)) / $timeDiff / 1024 / 1024);
     my $bpsIn = int((($BytesIn - $old_BytesIn) * (4*2)) / $timeDiff / 1024 / 1024);
     print "\t$bpsOut Mbps out ($old_MaxBpsOut), $bpsIn Mbps in ($old_MaxBpsIn)\n" if $DEBUG;

     # store new values
     $ini{$_}{"bytesOut"} = $BytesOut;
     $ini{$_}{"bytesIn"} = $BytesIn;
     $ini{$_}{"timestamp"} = $timeStamp;

     if ($max > $old_Max) {
          &tweet("$_ had a new Current Connections high @ $max (prev $old_Max)!");
          $ini{$_}{"maxconnections"} = $max;
     }

     if ($bpsOut > $old_MaxBpsOut) {
          &tweet ("$_ had a new Output Bandwidth High @ $bpsOut Mbps (prev $old_MaxBpsOut)!");
          $ini{$_}{"maxbpsOut"} = $bpsOut;
     }
     if ($bpsIn > $old_MaxBpsIn) {
          &tweet("$_ had a new Inbound Bandwidth High @ $bpsIn Mbps (prev $old_MaxBpsIn)!");
          $ini{$_}{"maxbpsIn"} = $bpsIn;
     }

     # update .ini
     tied( %ini )->WriteConfig( "zxtm.ini" ) ||     die "Could not write settings to new file.";

     print "\n" if $DEBUG;

     # add a line break for readability
     print "\n" if $DEBUG;

}

sub getMaxConn {
     my @vs = @_;
     my $total;
     my $res;

     $res = $conn1->getVirtualserverMaxConn(@vs); $total += $res->result->[0];
     $res = $conn2->getVirtualserverMaxConn(@vs); $total += $res->result->[0];
     $res = $conn3->getVirtualserverMaxConn(@vs); $total += $res->result->[0];
     $res = $conn4->getVirtualserverMaxConn(@vs); $total += $res->result->[0];
     $res = $conn5->getVirtualserverMaxConn(@vs); $total += $res->result->[0];

     $total;
}

sub getBytesOut {
     my @vs = @_;
     my $total;
     my $res;

     $res = $conn1->getVirtualserverBytesOut(@vs); $total += $res->result->[0];
     $res = $conn2->getVirtualserverBytesOut(@vs); $total += $res->result->[0];
     $res = $conn3->getVirtualserverBytesOut(@vs); $total += $res->result->[0];
     $res = $conn4->getVirtualserverBytesOut(@vs); $total += $res->result->[0];
     $res = $conn5->getVirtualserverBytesOut(@vs); $total += $res->result->[0];

     $total;

}

sub getBytesIn {
     my @vs = @_;
     my $total;
     my $res;

     $res = $conn1->getVirtualserverBytesIn(@vs); $total += $res->result->[0];
     $res = $conn2->getVirtualserverBytesIn(@vs); $total += $res->result->[0];
     $res = $conn3->getVirtualserverBytesIn(@vs); $total += $res->result->[0];
     $res = $conn4->getVirtualserverBytesIn(@vs); $total += $res->result->[0];
     $res = $conn5->getVirtualserverBytesIn(@vs); $total += $res->result->[0];

     $total;
}

sub tweet {
     my $message = shift;
     print "\ttweet: $message\n" if $DEBUG;

     my $browser = LWP::UserAgent->new;
     $browser->agent('ZXTM Eventing System');
     $browser->credentials(
          'twitter.com:80',
          'Twitter API',
          $tweetusr => $tweetpwd
);

     my $response = $browser->post( $updateURL, [ status => $message ] );

     #print $response->status_line;
     #print $response->content;

}

So it’s with a lot of excitement that I get to electronically introduce Shyam, who joins IT/Ops today. Shyam lives and breathes in Singapore and is our first non-US based IT/Ops member. He will make supporting our worldwide users easier and especially help support Europe/Asia in more real-time.

If you happen to live nearby, hopefully you’ll get a chance to meet him. For all those remote, I’m sure you’ll find him online at all the usual IT/Ops hangouts.

He knows that if he wants to get online he has to double-click on the Firefox icon and then on the Kidzui icon:

He asked me if I could come to his school and show people how I “fix computers”.

I started off by asking if anyone recognized the logo on the back of my shirt.  Even to a room full of 5 year olds, the Firefox logo was instantly recognizable – there wasn’t anyone who didn’t know what it was.  I talked briefly about what Mozilla did (”we make a web browser”) and that I help fix computers when they break.  

I thought about showing them Mitchell’s Mozilla Tree but probably couldn’t have done as well as Mitchell could have! 

Then I pulled out my laptop and showed them Firefox (Minefield really).  I was in the middle of showing them Firefox and how what my son does when he wants to get on the Internet… and Minefield crashed.  Which was a great segue into “lets go look at these computers I brought and how I fix them!”

I brought three old Celeron “servers” (you can hardly call a Celeron a server) with lids off and we spent the next 20 minutes taking apart the machines.  We took out the computer’s brain and the fan to keep it cool (they didn’t believe me that you could cook food on the CPU when the computer was “thinking hard”).  We took out the two memory sticks and the hard drive and the IDE cable.  

These are things most parents in their right mind wouldn’t do with their home computer and these kids really enjoyed physically touching these parts and asking questions.

Mary Colvig helped me gather up a bunch of Firefox bags and my two kids and I had stuffed stickers into each one. Kids went crazy over the Foxkeh stickers and there was a collective “awwww!” when I showed them the “don’t hurt the web” stickers!

Anyways, good times. I enjoyed talking about Mozilla in a very different setting than I’m used to.

I’m going to go work on my new role in community outreach now…

Over the past month we’ve worked out a different path than I originally thought we’d take.  Instead of targeting the Uninstall survey, I’ve been focusing my energies on point to explore #2:

2. How easily and quickly it is to get Firefox support using support.mozilla.com (is it too slow to be useful?)

I’ve been working closely with with David Tenser, Ken Kovash, and Seth Bindernagel (and all the l10n folks) to add a Kamplye survey to support.mozilla.com (SUMO). IT’s going to tag along with some survey questions David wanted to do. We hope to have something in place for June 1 (bug 489685).

I decided to target two major geographic regions (South America & Asia-Pacific) and have gotten help from our l10n community in translating the survey. You can follow our progress in bug 493096.

I’m including a sample mock-up below. I’m interested in any feedback you might have.

At the beginning of the quarter I set an ambitious goal of trying to look at ways IT/Ops can reduce Firefox adoption/retention barriers.

I kept rotating on the best way to frame this question and in the process probably kept over thinking it. Mentally I tied this goal into looking at under-served geographic regions like South America (Brazil) and Asia-Pacific. I’m trying to answer these questions:

1. How can IT help drive adoption/retention in Asia-Pacific or South America?
2. What sort of local IT resources does it take to move the needle?
3. If I put a data center in Brazil or Singapore, does it help?  Does it move the needle?  Do users care? (What’s my ROI?)

In other words, is there some function of Mozilla’s network or server infrastructure that prevents users in far away geographies from using Firefox?

I’ve been working with Asa Dotzler, Ken Kovash, Seth Bindernagel and Staś Małolepszy to try to figure out the right questions to ask and how to get them asked.

We came up with several points to explore:

1. Web page load time for Mozilla properties (www.mozilla.com/www.mozilla.org) & other “participatory sites” (planet.mozilla.org, bugzilla.mozilla.org, developer.mozilla.org, labs.mozilla.com…)
2. How easily and quickly it is to get Firefox support using support.mozilla.com (is it too slow to be useful?)
3. Interactive browsing on addons.mozilla.org (is this site too slow to use?)
4. Performance of AMO updates to extensions inside the Addons manager.

I had come up with three survey methods:

1. Installer survey
2. Uninstall survey
3. Community surveys

The Installer survey is interesting but I’m not convinced that helps answer my questions.  At the point of that survey, the user hasn’t really used Firefox and any of those websites.  So I scratched it.

My goal for Phase 1 is to target the Uninstall survey and relate the responses to the users geography.

Looking for suggestions – followups in comments or email.

During a normal release we have tools we can use to adjust the rate at which we offer updates. We use this to reduce load on the back end systems or to help reduce load on the download mirrors.

Our preference is to do a release completely unthrottled so users get timely updates.

During the Firefox 3.0.6 release we had a number of system problems that prevented us from releasing updates unthrottled. These were all detailed in the Post Mortem.

To the Operations Team’s credit (and I’m serious here), most of those issues were removed prior to yesterday’s Firefox 3.0.7 release and by 9am this morning we were cranking along – no throttling.

Unfortunately the Mirror Network started showing pressure and instead of throttling back on the release, we opted to augment the Mirror Network with our own download servers in San Jose.

That pushed our aggregate bandwidth out of San Jose to nearly 3Gbps:

At around this time offsite monitors starting alerting about a sharp increase in page load times to various Mozilla website properties. Took a bit to track down but the newly turned up Level 3 peer was saturated:

Any outbound traffic whose best route was out through Level3 was impacted. We fixed this temporarily by turning down Level3.

(I should note that our design requirements for upstream transit is at least two connections per provider so we can push 2Gbps. Level 3 is no exception, however, the second connection has been offline because Derek was seeing a lot of packet loss across the optical connection which coincidentally got resolved today.)

These problems are solvable and we’ve had plans to put tools in place to balance load during situations like this. Unfortunately, today’s issues came up a lot quicker than we had planned.

A couple things we’ll be looking at before the next release:

1. Evaluating Internap’s FCP to dynamically shift traffic based on cost and performance metrics. (And as luck would have it, this showed up this afternoon!)
2. Looking to see how we can better balance outbound traffic outside of using FCP.
3. Adding capacity to our Mirror Network (can you help?).
4. Evaluating options around upgrading from several 1GE upstream connections to 10GE connections.

This is a great problem to have, to be sure, and a far cry from the panic three years ago of “OMG we’re about to push 100Mbps!”.

I’m really interested in how others have gone about solving problems like this. Leave me comments.