Thursday, October 22nd, 2009
Mozilla’s North American store has been down in maintenance mode (you can read about the why) but not the International Store.
The old store used to redirect non-North American users to the International Store, but unfortunately that redirect wasn’t carried over when we took the store offline.
I was inspired Sunday night to put together a ZXTM TrafficScript rule based off the code sample on Zeus’s knowledgehub (and because of bug 521914).
And then on Wednesday Zeus finally released ZXTM 6.0 with geolocation support!
What was 40 some lines of code is now like 5.
Before:
$ipaddr = request.getRemoteIP();
# Integer representation of $ipaddr >> 1
string.regexmatch( $ipaddr, "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+)" );
$ip = ((($1*256+$2)*256+$3)*128+$4/2);
$arr = resource.get( "geoip.dat" );
# initialize indices
$i = 0; $j = string.len( $arr )/6-1;
# $arr[$i] <= $ip < $arr[$j]
# iteratively halve the distance between $i and $j until they are adjacent
while( $j-$i > 1 ) {
# midpoint between $i and $j
$k = ($i+$j)/2;
# compare $ip with $arr[$k]
if( string.bytesToInt( string.subString( $arr, $k*6, $k*6+3 ) ) > $ip ) {
$j = $k;
} else {
$i = $k;
}
}
# Now, $arr[$i] <= $ip < $arr[$j] and $j == $i+1
# Look up the 2-character country code (returns '??' if unknown)
$ccode = string.subString( $arr, $i*6+4, $i*6+5 );
if ( ($ccode != "US") && ($ccode != "CA") ) {
}
else {
http.redirect("https://intlstore.mozilla.org/");
}
After:
$ipaddr = request.getRemoteIP();
$country = geo.getCountryCode( $ipaddr );
if ( ($country != "US") && ($country != "CA") ) {
http.redirect("https://intlstore.mozilla.org/");
}
Friday, September 25th, 2009
(See Part V.)
I have a confession. We secretly did something last night (we only barely announced it to Metrics).
No, we didn’t secretly replace the fine coffee at some four-star restaurant but pretty close.
Hot off our 2 second gain in average page load times for addons.mozilla.org, we shaved another 2 seconds off by duplicating The Amsterdam Reboot platform in Singapore (as a proof-of-concept). Don’t take my word for it – take Gomez’s word:
A little background
For what seems like ages I’ve been trying to figure out how to best serve Asia-Pacific users. It’s a tough case to make because I didn’t have a method to easily measure how much bandwidth traffic I’d need or how it would change page load times or user perceptions.
But I’m a network guy and I’ve had this feeling that we need something in this region. We certainly have a growing population in the area – 5 of the top 9 countries we’ve seen > 20% growth in the last five months have been in Asia. I’ve only been lacking operational data.
Over the summer I’ve been playing and testing Voxel’s Silverlining Technology Preview, their global cloud computer platform. On October 1, it’s going to move out of Preview and their cloud platform would be generally available in all their POPs, including Singapore.
Seemed like a good way to get data…
In a matter of hours I had spun up two Zeus ZXTM and one GLB cloud servers with Voxel in Singapore. I waited till our normal Thursday night window before turning it live.
A couple take away notes on this
- We did it right. We built a proxy/caching platform as part of The Amsterdam Reboot that can easily be replicated anywhere and instantly provide real quantifiable performance benefits.
- Clouds make perfect sense to do proof-of-concepts.
- IT can move really fast. We had these three servers ready to go Wednesday morning in a couple hours..
- Mozilla’s webdev crew does an awesome job writing extremely cachable webapps. I’m seeing a 91% cache hit rate (350,000 objects, 1.5GB).
- If this is a sustainable location, pushing user-focused sites like
support.mozilla.com to Singapore are next on my list.
- I’d love to run this concept in other geographies like South America. Who does clouds down there?
What I most like about this platform is that it’ll allow us to strategically get content closer to users where it most makes sense.
This is right now just a proof-of-concept. It lets me experiment and get real metrics. I’m very interested in hearing from people who actually live in the area – does this make you happier?
One more thing
I was lucky to have two providers who stepped in and provided resources to let me run this POC. Both deserve a special thanks.
- Zeus. These guys are great and quickly issued temporary licenses for both ZXTM and GLB.
- Voxel. You guys have a nice platform in the right part of the world. I like working with you.
Friday, September 18th, 2009
2 seconds.
That’s the amount of time we shaved off average page load times for addons.mozilla.org after last night’s work. The Amsterdam Reboot in effect!
It’s been nearly ten months since we served production traffic at these levels out of Amsterdam.
Monday, August 24th, 2009
Three years ago this coming December I went to Amsterdam and installed our first non-US data center location.
I remember coming back and was up late at night (fighting jetlag) setting up the Netscaler load balancers. By early January we had a CVS mirror up and running and a week later had staged www.mozilla.com and www.mozilla.org in Amsterdam. By the middle of February we had shifted European production traffic over to Amsterdam.
By May of 2007 (oh and here) we started serving addons.mozilla.org out of Amsterdam too.
Since then, the San Jose data center has grown from seven racks to twenty-four and nearly 500 servers (for the sake of this post I’m counting the 150 Mac Minis as “servers”).
Unfortunately, Amsterdam hasn’t seen the same sort of server growth and in the past half year or so we’ve had to pull sites back from Amsterdam and serve them from San Jose only. When the load balancers there could no longer handle the SSL traffic in January we stopped serving addons.mozilla.org out of Amsterdam too.
The Reboot
On September 2, we’ll reboot Amsterdam. Much like The Six Million Dollar Man, “… we can rebuild [it]. We have the technology… We can make [it] better than [it] was before. Better, stronger, faster.”
This morning we shipped a half loaded HP c7000 BladeSystems chassis out to Amsterdam. Next week both Arzhel and Derek will be in Amsterdam to deploy new servers and turn down some of the old legacy hardware.
One of the issues with the current Amsterdam deployment is that we can only really serve static sites – sites that don’t have a database behind them. It’s a bit more complicated to replicate databases to remote locations and keep them in sync with San Jose.
Based on our success with the Zeus ZXTM platform, we’re deploying a Nehalem based ZXTM cluster. Much like we originally did with addons.mozilla.org, we’ll have a platform where we can proxy/cache any Mozilla website and serve it out of Amsterdam.
We’ll also be deploying a new global load balancing system that Arzhel’s spent time staging and getting ready for production.
The Plan
During tomorrow night’s downtime window we’ll start the process of temporarily shutting down Amsterdam. We’ll pull back all websites to San Jose and stop serving web content out of Amsterdam for about two weeks.
We’re doing this in advance of the actual deployment to make sure we don’t hit any surprises.
What won’t be affected?
There are two VMware ESX servers in Amsterdam that we won’t be touching. A quick list of hosts that won’t be affected by this are:
- dn-vcs01
- gravel
- geodns02
- l10n-01
- rhino01
- sea-qm-centos5-01
- sea-qm-win2k3-01
- konigsberg
I’m excited about this. I’ve said before I’m still a network guy at heart and I really like getting content closer to consumers. I’m hoping this Reboot becomes an easily deployable platform for other parts of the globe.
Saturday, June 27th, 2009
I was recently inspired by morgamic’s Org Chart coding exercise and that got me thinking about an article I read some months ago about Zeus’ ZXTM triggers.
After a couple hours this week re-learning Perl and learning SOAP I have something that I think is cool - @mozdashboard!
I experimented a bit earlier in this week before Zandr gave me an idea of what I was really looking for with this. This version keeps state using Config::IniFiles (yeah, I said it and it was surely an easy way out) and tweets when it detects new highs.
zxtmtwitter.pl tracks the following sites:
addons.mozilla.orgversioncheck.addons.mozilla.orgfxfeeds.mozilla.com
It’ll tweet if it detects a new bandwidth high (either inbound or outbound) or a new simultaneous current connection count.
The program’s not without some faults – I’m not accounting for any counter wraps and I’m not happy with some of the hackery to poll each ZXTM node (I can’t find a way to poll one and get the cluster aggregates).
Code and .ini below the fold.
Continued reading >
10:27 < mrz> are we -sure- we're in the middle of a 3.0.11 release?
10:27 < reed> yes, #mirrors is lively
10:27 < mrz> i don't see anything melting down
Although Mozilla’s IT/Ops Team supports a worldwide user base, we’re all located in North America. That’s increasingly become a challenge as we have more employees, l10n folk and users living outside of North America.
So it’s with a lot of excitement that I get to electronically introduce Shyam, who joins IT/Ops today. Shyam lives and breathes in Singapore and is our first non-US based IT/Ops member. He will make supporting our worldwide users easier and especially help support Europe/Asia in more real-time.
If you happen to live nearby, hopefully you’ll get a chance to meet him. For all those remote, I’m sure you’ll find him online at all the usual IT/Ops hangouts.
Wednesday, May 20th, 2009
My son has a vague concept of what I do at work. He knows I work at “Firefox”, knows the dino and knows I “fix computers”.
He knows that if he wants to get online he has to double-click on the Firefox icon and then on the Kidzui icon:
He asked me if I could come to his school and show people how I “fix computers”.
I started off by asking if anyone recognized the logo on the back of my shirt. Even to a room full of 5 year olds, the Firefox logo was instantly recognizable – there wasn’t anyone who didn’t know what it was. I talked briefly about what Mozilla did (”we make a web browser”) and that I help fix computers when they break.
I thought about showing them Mitchell’s Mozilla Tree but probably couldn’t have done as well as Mitchell could have!
Then I pulled out my laptop and showed them Firefox (Minefield really). I was in the middle of showing them Firefox and how what my son does when he wants to get on the Internet… and Minefield crashed. Which was a great segue into “lets go look at these computers I brought and how I fix them!”
I brought three old Celeron “servers” (you can hardly call a Celeron a server) with lids off and we spent the next 20 minutes taking apart the machines. We took out the computer’s brain and the fan to keep it cool (they didn’t believe me that you could cook food on the CPU when the computer was “thinking hard”). We took out the two memory sticks and the hard drive and the IDE cable.
These are things most parents in their right mind wouldn’t do with their home computer and these kids really enjoyed physically touching these parts and asking questions.
Mary Colvig helped me gather up a bunch of Firefox bags and my two kids and I had stuffed stickers into each one. Kids went crazy over the Foxkeh stickers and there was a collective “awwww!” when I showed them the “don’t hurt the web” stickers!
Anyways, good times. I enjoyed talking about Mozilla in a very different setting than I’m used to.
-
-
Who knows this logo?
-
-
Minefield crashed with Kidzui
-
-
Inside a computer
-
-
And this is a hard drive
I’m going to go work on my new role in community outreach now…
Nearly a month and a half ago I posted my first post on trying to figure out how IT can reduce adoption/retention barriers.
Over the past month we’ve worked out a different path than I originally thought we’d take. Instead of targeting the Uninstall survey, I’ve been focusing my energies on point to explore #2:
2. How easily and quickly it is to get Firefox support using support.mozilla.com (is it too slow to be useful?)
I’ve been working closely with with David Tenser, Ken Kovash, and Seth Bindernagel (and all the l10n folks) to add a Kamplye survey to support.mozilla.com (SUMO). IT’s going to tag along with some survey questions David wanted to do. We hope to have something in place for June 1 (bug 489685).
I decided to target two major geographic regions (South America & Asia-Pacific) and have gotten help from our l10n community in translating the survey. You can follow our progress in bug 493096.
I’m including a sample mock-up below. I’m interested in any feedback you might have.
Tuesday, March 24th, 2009
Why don’t you use Firefox and can IT change that?
At the beginning of the quarter I set an ambitious goal of trying to look at ways IT/Ops can reduce Firefox adoption/retention barriers.
I kept rotating on the best way to frame this question and in the process probably kept over thinking it. Mentally I tied this goal into looking at under-served geographic regions like South America (Brazil) and Asia-Pacific. I’m trying to answer these questions:
- How can IT help drive adoption/retention in Asia-Pacific or South America?
- What sort of local IT resources does it take to move the needle?
- If I put a data center in Brazil or Singapore, does it help? Does it move the needle? Do users care? (What’s my ROI?)
In other words, is there some function of Mozilla’s network or server infrastructure that prevents users in far away geographies from using Firefox?
I’ve been working with Asa Dotzler, Ken Kovash, Seth Bindernagel and Staś Małolepszy to try to figure out the right questions to ask and how to get them asked.
We came up with several points to explore:
- Web page load time for Mozilla properties (
www.mozilla.com/www.mozilla.org) & other “participatory sites” (planet.mozilla.org, bugzilla.mozilla.org, developer.mozilla.org, labs.mozilla.com…)
- How easily and quickly it is to get Firefox support using
support.mozilla.com (is it too slow to be useful?)
- Interactive browsing on
addons.mozilla.org (is this site too slow to use?)
- Performance of AMO updates to extensions inside the Addons manager.
I had come up with three survey methods:
- Installer survey
- Uninstall survey
- Community surveys
The Installer survey is interesting but I’m not convinced that helps answer my questions. At the point of that survey, the user hasn’t really used Firefox and any of those websites. So I scratched it.
My goal for Phase 1 is to target the Uninstall survey and relate the responses to the users geography.
Looking for suggestions – followups in comments or email.
