addons.mozilla.org was updated last week. AMO 3.2.1 was a maintenance release (26 bugs fixed) for any major issues with 3.2.
Our next release will be AMO 3.4.1, the first of three dot releases for AMO 3.4, which is our next milestone to be completed before Firefox 3.
We’ve pushed some important updates in the last couple of days:
This means:
Thanks for everyone’s patience with the crash report backlog during releases — we hope this helps many of you.
Let me know if you have any questions. More to come in the next few weeks! Thanks to Lars, Ted and Aravind for their help with developing/ testing and pushing these updates.
Ran statsvn on r7797 through r11622. View svn stats for AMO 3.2.
All week, all the time:
On the developers page, you will see how many non-mozilla contributors have helped us update translations. Overall, some rough stats:
Again, thanks a ton for your help!
Yep — “sexy” and “AMO” in the same sentence. Our festively plump 3.2 target milestone added many features and refreshes that you can read about in a great post on Basil’s Bodacious blog.
In the same time period we’ve also:
And really, when it comes to sexy, the real magic starts with our volunteers. Our editors have worked hard to review new and updated add-ons as we move towards Firefox 3 this year and our localizers translated roughly 200 new strings in AMO templates in a little over three weeks for 24 locales (wow).
Thanks to everyone for pitching in to make this release happen.
For about 48 hours the AMO API was thrashing because of the popularity (and hunger!) of the new add-ons manager in Firefox 3 Beta 3. The dust has settled and the servers are humming happily along, so now is a good time to blog about what happened and how we’ll handle future releases successfully.
Stop. Take a deep breath. Alright, here we go.
Now that the API is functional (most major bugs have been ironed out) we got a rude awakening this week and found out exactly how much traffic the improved Add-ons Manager can generate, but it’s a nice problem to have and we’re happy it’s been well received.
Wednesday, around peak time, the API started clobbering our databases:
Shortly after we entered our peak traffic window, we had to turn off the API to keep the normal AMO working. Diagnosis found that:
Wednesday, IT and Webdev spent quite a bit of time getting the API back up. Starting with the three points above, we:
However, Thursday didn’t fare any better for the cluster. This time the slaves started to melt near peak time — forcing us to once again temporarily disable the API. Under-utilizing memcache was the main issue. Cache headers were fine, slave was utilized, app nodes were fine — just too many damn queries flying at our database servers! 
So on Thursday we continued our look into what was going on. We tried to figure out why our cache hit rate was so low (60% instead of 90%). Digging through AMO, we found CACHE_PAGES_FOR, which set the expire time on memcache records when calling Memcache::set(), was set to 60 seconds. We increased this to 7200 to aggressively cache database traffic and were collectively off for valentine’s dinner.
The next day, Memcache was our valentine.
The combination of our efforts worked:
So these growing pains will help us move forward. Here is our plan of attack for scaling this beast for the Firefox 3 onslaught:
Once again it was a great team effort to get things running smoothly. Thanks to IT for helping us troubleshoot this. We’ll continue to build on this experience to ensure better reliability in future releases.
Looking back at the last three days, the Firefox 3 Beta 3 release was a success in more ways than one. It showed everyone what the web can do, but it also helped us wrap our heads around the API and how much traffic it generates. All of this will make for a better Firefox 3.0 release.
AMO has a new look and we need your help to polish it off. Please tell us what you think!
Here are some screenshots:
Aside from a new look, here are few highlights in AMO 3.2:
What we’d like to know:
Keep in mind that we are still ironing out some wrinkles. For more information:
Thanks, and looking forward to hearing from everyone.
Yes, we are over 10,000 commits in our subversion repository. This last update for AMO trunk includes the following fixes, among others:
I want to thank everyone on the AMO team for their hard work, especially localizers who have worked really hard to port AMO to their native language. 2008 is already turning out to be a great year — let’s keep it up!
I was looking at one of the AMO v3.2 mockups today. There are strings like “See All Interface Tweaks Add-ons” that we’ve avoided up till now, but this isn’t the first time they’ve been proposed. The problem we’re having is that a string like that is from two different sources - static and dynamic data. “Interface Tweaks” is the name of one of our categories so it’s stored in the database, and the rest of the string is static, so it’s in a .po. The static string would look something like:
See All %s Add-ons
and the dynamic string would look like:
Interface Tweaks
In English, these combine and all is well, but if the second value affects the structure of the first in a dynamic way, we can’t support the phrase on AMO.
When I wrote the original code, I think I had two things in mind:
Categories have changed a bit in the past (and they’re in mid change right now, actually), but other than the convenience of a near-instant change on the site, it doesn’t seem that beneficial to change them via the web. The second point is a big one though - by giving localizers direct access to SVN, they can update strings whenever they need to without our meddling and getting in the way. That’s a big time saver for everyone.
So, now I’m reconsidering the separation of some of the interface translations (add-on types, applications, and categories) from the rest of the static content. Looking at where we currently stand, it seems like it’s more of a hassle to describe the separation and what it does than it would be to just drop everything in the .po file. Plus we’d get the benefit of strings like “See All Interface Tweaks Add-ons.”
Recently, I worked on a nice little feature for AMO: Letting developers reply to user reviews.
The idea is, when you get a review as an add-on publisher, you may find a spot or two in it that you feel like replying to. In the previous version of AMO, users started discussions by just adding another review (with a random rating), effectively rendering parts of the rating system useless. Also, the developers were not allowed to rate their own add-ons and could thus not reply to any of the questions.
In AMO version 3, we addressed this issue by having our editor team moderate reviews to ensure “good” reviews and thus useful ratings. For discussions around individual add-ons, we introduced a forum system to provide a more appropriate means of discussing support questions and similar while not diluting the review/rating system. This, however, left reviews as a one-way communication with no way for developers to address questions raised in their add-ons’ reviews.
This issue is now fixed: As of today’s AMO update, developers can now reply once to each of the published reviews of their add-ons.
The reason why we only allow one reply is so that discussions are held in the discussion forums (which is where they should be, as mentioned above), while not forcing developers to keep possible allegations undisputed. Some of you may notice parallels to the rating system Ebay uses for sellers.
Before you start replying to a bunch of reviews now: Remember, the reviews of your add-on are a place for your users to give feedback and point out good or bad things (in their opinion) about your add-on. Developer replies will also be moderated by editors, just like regular reviews. So, as always, play nice, even if you disagree with the opinion explained in the user review. If you get a bad review once, don’t fret: Rather than “yelling” at the user (which will most likely be deleted by an editor anyway), you should take it as an encouragement to make your add-on better. After all, the user considered your add-on interesting enough to download and test it and give you feedback on the points they found improvable. This is how great add-ons start!
We hope you enjoy the new feature! If you find any problems with it, feel free to drop by #amo on irc.mozilla.org or to file a bug as usual.
Bouncer had a few updates last night:
To see the difference, I’ll show a couple of database graphs. Note that logging is still turned on, so there are still database connections used for updating product and mirror counts. We plan on disabling this in the future because these stats are backfilled from HTTP logs.
Pretty CPU graph:
Pretty traffic graph:
So CPU and traffic data was reduced significantly. Here’s why:
Again, database traffic flatlined during testing when logging was disabled so what you’re seeing is basically download count updates and the connections needed to run them.
To get an idea of exactly how much memcache is used, here are stats from memcache as of this morning (single server, after approximately 12 hours of usage):
This means that of all queries, 99.33% were read from the cache. Not bad.
This patch took two weeks longer than expected. Challenges we faced:
Overall, this is another positive experience with memcache as a query cache. We use the same method in AMO and plan on using it in other apps that will have to scale to any reasonable level. It saves money in hardware and turns little apps that can sort-of do the job into supercharged workhorses. Yay.
