Everybody knows Mozilla makes Firefox. But there is a lot more software at work here at Mozilla that you might not be aware of. For example: What happens when you go to getfirefox.com and click on the download button?

By clicking on the button, you ask our servers to send you a specific file, for example: Firefox 3.6.3, for Windows, in German. On a small website, the server would just fetch the file and hand it to you. But if you need to handle millions of downloads a day like we do, a single server can’t handle it all by itself, so it gets more complicated. In order to provide you with downloads, updates, etc., as fast and conveniently as possible, Mozilla collaborates with a number of mirror providers that have volunteered to host Firefox and other downloads on our behalf, thus sharing the load of our numerous downloads between a number of servers all over the world.

For some years now, we have been running a bundle of software called “Bouncer” to handle our downloads for us.

Bouncer consists of of three components: The user-facing bounce script, an administrative interface called Tuxedo, and a mirror checker called Sentry.

First, the bounce script. It is the only component the “ordinary user” gets to interact with. It essentially does the following after you click on a download link:

• It determines if the product you asked for exists.
• Out of our list of mirrors, it picks one that has your file. Initially, it would pick one at random. Over the years, the logic has become more elaborate though: Meanwhile, it takes into account in what country you currently are, as well as how strong the mirrors are (stronger mirrors serve more downloads, weaker ones serve less).
• A split-second later, Bouncer refers you to the server it decided on, and that server will send you the file you asked for.

But wait, there is more! How does Bouncer know what products are available, for what operating systems, and in what languages? That’s where the admin interface comes in. We have a release engineering team who work hard every day to deliver the newest software versions to you in handy little packages. Previously, during every release, an engineer would manually tell Bouncer that a new version was available for download. But just last week, we improved this process by introducing a new interface to Bouncer, with a project called Tuxedo. The release engineering team can now, fully automatically, feed new versions into Bouncer at the time of release, with no manual intervention. With less time spent on repetitive tasks, we can spend more time making Firefox awesome.

Finally, the Sentry component is a script that periodically checks the health of our mirrors, and adjusts our settings accordingly. This is to ensure that a situation where you are forwarded to a mirror that is currently unavailable is very, very rare. So far, these mirror checks happen from Mozilla Headquarters, and therefore reflect the connectivity we get to the mirrors from here. In the future, we want to improve that by taking into account more how our users’ connectivity is to the specific mirrors (for the geeks out there: Network proximity != geographical proximity), which has the potential to result in faster download times, less expenses for mirror providers, and general happiness.

As you can see, there are a lot of things happening behind the scenes before Firefox makes its way onto your computer at home, and we are constantly working on improving the way we are doing things. Plus, as always: Bouncer is completely open source, and we have a public bug tracker, so if you notice any problems or see room for improvement, make sure to let us know.

Photo credit: “directions”, CC-by licensed by Phillie Casablanca.

• Database library rewritten to be lightweight and use memcache (see tests)
• Overall requests per second performance increased 2x
• Database usage down to about zero because of memcache

To see the difference, I’ll show a couple of database graphs. Note that logging is still turned on, so there are still database connections used for updating product and mirror counts. We plan on disabling this in the future because these stats are backfilled from HTTP logs.

Pretty CPU graph:

Pretty traffic graph:

So CPU and traffic data was reduced significantly. Here’s why:

• Bouncer is a special type of app – high amount of reads over a short period of time
• The majority of these database reads are repeated over and over, especially recent releases (Firefox 2.0.0.6 for example)

Again, database traffic flatlined during testing when logging was disabled so what you’re seeing is basically download count updates and the connections needed to run them.

To get an idea of exactly how much memcache is used, here are stats from memcache as of this morning (single server, after approximately 12 hours of usage):

• Gets: 29884170
• Misses: 199515
• Total Gets: 30083685
• Hit %: 99.33679999641

This means that of all queries, 99.33% were read from the cache. Not bad.

This patch took two weeks longer than expected. Challenges we faced:

• Getting random mirror selection to work when not using “ORDER BY RAND()” as the way to randomize mirror selection — this was replaced by a simple hash and array sort algorithm that uses array values to weight overall probability of selection when combined with a random seed. In general, random selection via SQL is inefficient and you’re better off using another method besides just RAND() — stored procedures, app code, whatever — but when you’re doing “ORDER BY RAND()” you’re in for some pain. Use EXPLAIN — it does not lie.
• Being stingy with MySQL connections. The database library rewrite was required because we did not want to even connect to the database when memcache had all of our results cached. However, mysql_real_escape_string() was used to escape queries and it requires an open database handle to work. That meant that we had to only escape if we knew we were going to perform a query. So we had to move cleaning of SQL inputs to inside the query callback function. This was done by…
• Breaking away from the PHP4 mysql habit. PHP 5.x has better support for this via mysqli but since we’re still on PHP4 for a little while, in order to mimic prepared statements without using a database interface we used a method much like what was discussed among WordPress developers. It takes us from a connect() ->clean -> concatenate sql -> query() approach to a prepared sql and args -> query() -> autoclean approach.

Overall, this is another positive experience with memcache as a query cache. We use the same method in AMO and plan on using it in other apps that will have to scale to any reasonable level. It saves money in hardware and turns little apps that can sort-of do the job into supercharged workhorses. Yay.