We attempted to push the AMO 3.4.3 update last night but ran into some problems. Our initial investigation is pointing to the advanced search queries taking far longer than the previous version. The slave databases couldn’t handle the load and we had to back the update out. We’ll come up with a solution this weekend and plan on pushing the 3.4.3 update on June 12th.
As previously mentioned we’re planning on updating addons.mozilla.org for the 3.4.2 changes tonight. There are 32 bugs that will be fixed with the update.
Please let us know if you see any regressions or anything you’d like to see changed.
The next update is scheduled for May 29th and will be mainly a bug fix release with minor new features (like bug 432669). There are currently 28 bugs scheduled for the 3.4.3 push.
The AMO team is finalizing the latest update, 3.4.1. This is a bug-fix release addressing things like double escaping, improper L10n redirects, and category cleanup. A list of all the bugs targeted for this release is available.
We’ll be committing the last of the patches very shortly and the changes will be available on preview.addons.mozilla.org. Feel free to look at the changes and send us any feedback you have. We expect to push the changes live this Thursday evening (May 1st).
Our next release, 3.4.2, is currently scheduled for May 15th. 3.4.2 will also be a bug-fix release - there are currently 19 candidates.
addons.mozilla.org was updated last week. AMO 3.2.1 was a maintenance release (26 bugs fixed) for any major issues with 3.2.
Our next release will be AMO 3.4.1, the first of three dot releases for AMO 3.4, which is our next milestone to be completed before Firefox 3.
Ran statsvn on r7797 through r11622. View svn stats for AMO 3.2.
All week, all the time:
On the developers page, you will see how many non-mozilla contributors have helped us update translations. Overall, some rough stats:
Again, thanks a ton for your help!
Yep — “sexy” and “AMO” in the same sentence. Our festively plump 3.2 target milestone added many features and refreshes that you can read about in a great post on Basil’s Bodacious blog.
In the same time period we’ve also:
And really, when it comes to sexy, the real magic starts with our volunteers. Our editors have worked hard to review new and updated add-ons as we move towards Firefox 3 this year and our localizers translated roughly 200 new strings in AMO templates in a little over three weeks for 24 locales (wow).
Thanks to everyone for pitching in to make this release happen.
For about 48 hours the AMO API was thrashing because of the popularity (and hunger!) of the new add-ons manager in Firefox 3 Beta 3. The dust has settled and the servers are humming happily along, so now is a good time to blog about what happened and how we’ll handle future releases successfully.
Stop. Take a deep breath. Alright, here we go.
Now that the API is functional (most major bugs have been ironed out) we got a rude awakening this week and found out exactly how much traffic the improved Add-ons Manager can generate, but it’s a nice problem to have and we’re happy it’s been well received.
Wednesday, around peak time, the API started clobbering our databases:
Shortly after we entered our peak traffic window, we had to turn off the API to keep the normal AMO working. Diagnosis found that:
Wednesday, IT and Webdev spent quite a bit of time getting the API back up. Starting with the three points above, we:
However, Thursday didn’t fare any better for the cluster. This time the slaves started to melt near peak time — forcing us to once again temporarily disable the API. Under-utilizing memcache was the main issue. Cache headers were fine, slave was utilized, app nodes were fine — just too many damn queries flying at our database servers! 
So on Thursday we continued our look into what was going on. We tried to figure out why our cache hit rate was so low (60% instead of 90%). Digging through AMO, we found CACHE_PAGES_FOR, which set the expire time on memcache records when calling Memcache::set(), was set to 60 seconds. We increased this to 7200 to aggressively cache database traffic and were collectively off for valentine’s dinner.
The next day, Memcache was our valentine.
The combination of our efforts worked:
So these growing pains will help us move forward. Here is our plan of attack for scaling this beast for the Firefox 3 onslaught:
Once again it was a great team effort to get things running smoothly. Thanks to IT for helping us troubleshoot this. We’ll continue to build on this experience to ensure better reliability in future releases.
Looking back at the last three days, the Firefox 3 Beta 3 release was a success in more ways than one. It showed everyone what the web can do, but it also helped us wrap our heads around the API and how much traffic it generates. All of this will make for a better Firefox 3.0 release.
AMO has a new look and we need your help to polish it off. Please tell us what you think!
Here are some screenshots:
Aside from a new look, here are few highlights in AMO 3.2:
What we’d like to know:
Keep in mind that we are still ironing out some wrinkles. For more information:
Thanks, and looking forward to hearing from everyone.
Yes, we are over 10,000 commits in our subversion repository. This last update for AMO trunk includes the following fixes, among others:
I want to thank everyone on the AMO team for their hard work, especially localizers who have worked really hard to port AMO to their native language. 2008 is already turning out to be a great year — let’s keep it up!
I was looking at one of the AMO v3.2 mockups today. There are strings like “See All Interface Tweaks Add-ons” that we’ve avoided up till now, but this isn’t the first time they’ve been proposed. The problem we’re having is that a string like that is from two different sources - static and dynamic data. “Interface Tweaks” is the name of one of our categories so it’s stored in the database, and the rest of the string is static, so it’s in a .po. The static string would look something like:
See All %s Add-ons
and the dynamic string would look like:
Interface Tweaks
In English, these combine and all is well, but if the second value affects the structure of the first in a dynamic way, we can’t support the phrase on AMO.
When I wrote the original code, I think I had two things in mind:
Categories have changed a bit in the past (and they’re in mid change right now, actually), but other than the convenience of a near-instant change on the site, it doesn’t seem that beneficial to change them via the web. The second point is a big one though - by giving localizers direct access to SVN, they can update strings whenever they need to without our meddling and getting in the way. That’s a big time saver for everyone.
So, now I’m reconsidering the separation of some of the interface translations (add-on types, applications, and categories) from the rest of the static content. Looking at where we currently stand, it seems like it’s more of a hassle to describe the separation and what it does than it would be to just drop everything in the .po file. Plus we’d get the benefit of strings like “See All Interface Tweaks Add-ons.”
