The Mozilla community is blessed with a lot of freely donated bandwidth through our mirror network. This network handles product downloads through bouncer and a subset of this handles releases.mozilla.org, which carries contains Firefox, Thunderbird, Camino, Seamonkey, and Sunbird releases and Add-ons (and is huge, clocking in around 100-130G).
releases.mozilla.org is handled through round-robin DNS. Regardless where you are in the world, you’ll get a “random” release mirror that may or may not be anywhere close to you and because of various physical reasons might induce a lot of latency and may or may not take a long time to download (admittedly, non-interactive transfers are less worried about latency but sitting in New York and downloading from China is going to take a long time).
As a network guy, I like to get my content as close to users as possible. I know from my two visits to China that getting Add-ons updates from European mirror blow especially when I’m mere miles from our China release mirror!
After looking at a couple commercial solutions we decided to build our own geodns solution using the latest version of BIND and two patches to add Maxmind’s GeoIP functionality and to replace the backend zonefiles with MySQL. That last patch lets us do all sorts of interesting thinks like nailing off bug 406267 (easily anyways, without having to munge text files).
Mark’s been working on all of the CLI management end to enable us to add or disable mirrors. Hopefully I can get him to blog about the technical details but I’ll just add that prior to this I didn’t know anything about SQL views.
So when are we going deploy this? Glad you asked.
On Tuesday, May 27 we’ll flip the switch and change releases.mozilla.org to a CNAME to releases.geo.mozilla.com, which, until Tuesday, will be in various states of testing but should work should be so inclined to try.
Currently, of the 15 release mirrors, 9 are in the US and 2 are in China. The initial rollout will only directly benefit our Chinese users by directing them to the two Mozilla Online release mirrors in China. Later we’ll try to do something on a geo-regional basis (but first, we’d want more than 3 mirrors in EU- anyone?)
This is just the tip of the iceberg. Globally we use Netscaler-based GSLB to load balance and direct users to the network-wise closest site. However, not all Mozilla properties are behind the Netscaler (or need to be) and we haven’t had a good way to serve users from different data centers unless they were.
Down the road we planning on adding some sort of weighting (not all mirrors or sites are equal), query statistics (where are lookups coming from?) and some sort of Nagios integration to automatically disable/enable a record.
And hopefully users will rejoice!
