We’ll be committing the last of the patches very shortly and the changes will be available on preview.addons.mozilla.org. Feel free to look at the changes and send us any feedback you have. We expect to push the changes live this Thursday evening (May 1st).

Our next release, 3.4.2, is currently scheduled for May 15th. 3.4.2 will also be a bug-fix release - there are currently 19 candidates.

• Bug 426940 - Reduce or eliminate delay in collector to monitor hand-off
• Bug 426940 - Fix processor handling of error conditions
• Bug 428300 - status page too slow

This means:

• When you submit a crash report you won’t have to wait longer than 30-60 seconds to view your report
• The processor now has better handling of minidump_stackwalk fatal errors
• There is an improved server status page where you can view stats on the current queue

Thanks to Lars and Aravind for getting this out the door.  The next couple of weeks will be spent improving reporter performance and closing out milestone 0.5 bugs.

Mozilla doesn’t.

Rather than being the exception, openness is the rule, and that is one of the coolest things about being a part of this. My job, my everday tasks, they aren’t secret, and they are not to drive profits. They are to drive the web.

In that spirit, our crash reporting system (Socorro) is available to whoever wants to view it. Aside from user-bound statistics, crash information is available in full and anybody in the community can learn about where in the code their client crashed. They can also help provide hints or comments about what they were doing at the time they crashed.

This opens the door for the community to learn valuable things about their software and how they use it:

• What crashes the most? What crashes the most over time? What is the breakdown across branches, versions and products?
• Where did we crash? Crash signatures provide a head start for locating the cause for a crash. From there, full stack traces are available to analyze callback and find the source of the actual crash.
• What was installed? What modules were installed for a given crash? Soon we will also be able to understand what extensions were installed so we can understand the correlation between core client crashes and crashes caused by faulty extensions. The end result is a closer relationship with the extension developer community and better quality in our add-ons space.
• How are we doing? Overall the jackpot question is — are we crashing more or less? How are we doing with this beta, alpha or rc1? Are we regressing in real-life situations despite positive automated testing results??

All of this was possible because of a collaborative effort between quite a few parties:

• Mark Mentovai and the breakpad team, for writing a great client and processor under a flexible open source license that is easy to integrate
• Ted Mielczarek for his work on the client, processor and integrating the project into Firefox 3
• Benjamin Smedberg and Robert Sayre for their work in getting the initial versions of the breakpad server off the ground

Where do we go from here?

• Learn more about the project
• Send in your ideas

Of the many projects we have in 2008, this is one of the most exciting. It’s an opportunity to open up information that hasn’t historically been available to the masses, and hack on a great tool for improving the quality of all Mozilla projects

Our next release will be AMO 3.4.1, the first of three dot releases for AMO 3.4, which is our next milestone to be completed before Firefox 3.

• refactor of processor code, which is 1/3 of the breakpad server
  architecture
• update of reporter to allow for instant queuing of requested reports

This means:

• If you submit a crash, going to that crash page will:
  • Show you a “haven’t queued it yet” page instead of a 404
    page that will update in < 10 min
  • Once queued, you’ll see a “report pending” page that will
    redirect to the finished report in < 21 seconds
• Wait time for reports from testers is reduced to 10 min max,
  sometimes 21 seconds best-case
• We are working on eliminating the 10 min portion but there are
  reasons why we can’t spam the monitor that is responsible for
  queuing new reports that are on disk — more on that next week (I
  want this to get down to: load, wait 20 seconds, BAM! see your report)

Thanks for everyone’s patience with the crash report backlog during releases — we hope this helps many of you.

Let me know if you have any questions.  More to come in the next few weeks!  Thanks to Lars, Ted and Aravind for their help with developing/ testing and pushing these updates.

All week, all the time:

On the developers page, you will see how many non-mozilla contributors have helped us update translations.  Overall, some rough stats:

• 3134 changes
• 288602 lines affected
• 24 contributors

Again, thanks a ton for your help!

In the same time period we’ve also:

• Upgraded CakePHP’s core software from 1.18.x to 1.19.x
• Added support for weighted database slaves
• Migrated AMO to PHP5 off of the now end-of-life’d PHP4
• Replaced Scriptaculous/Prototype with jQuery
• Improved IE6/IE7 compatibility
• Improved accessibility features

And really, when it comes to sexy, the real magic starts with our volunteers.  Our editors have worked hard to review new and updated add-ons as we move towards Firefox 3 this year and our localizers translated roughly 200 new strings in AMO templates in a little over three weeks for 24 locales (wow).

Thanks to everyone for pitching in to make this release happen.

• What happened this week, anyway?
• How was it fixed?
• How are we going to scale for Firefox 3?

Stop.  Take a deep breath.  Alright, here we go.

What happened this week?

Now that the API is functional (most major bugs have been ironed out) we got a rude awakening this week and found out exactly how much traffic the improved Add-ons Manager can generate, but it’s a nice problem to have and we’re happy it’s been well received.

Wednesday, around peak time, the API started clobbering our databases:

Shortly after we entered our peak traffic window, we had to turn off the API to keep the normal AMO working.  Diagnosis found that:

• Load was not utilizing the read-only slave and was focused mainly on the master read/write database (mrdb03).
• Cache hit rates were down to 60% from the usual 90% for memcached
• When our databases hit peak CPU, the app cluster would tumble because of the piling requests

How it was fixed?

Wednesday, IT and Webdev spent quite a bit of time getting the API back up.  Starting with the three points above, we:

• Off-loaded read-only traffic to DB slaves
• Investigated optimizations for the API
• Looked at cache rules and cache policies for both memcache and the hardware load balancer

However, Thursday didn’t fare any better for the cluster.  This time the slaves started to melt near peak time — forcing us to once again temporarily disable the API.  Under-utilizing memcache was the main issue.  Cache headers were fine, slave was utilized, app nodes were fine — just too many damn queries flying at our database servers!

So on Thursday we continued our look into what was going on.  We tried to figure out why our cache hit rate was so low (60% instead of 90%).  Digging through AMO, we found CACHE_PAGES_FOR, which set the expire time on memcache records when calling Memcache::set(), was set to 60 seconds.  We increased this to 7200 to aggressively cache database traffic and were collectively off for valentine’s dinner.

The next day, Memcache was our valentine.

The combination of our efforts worked:

• Overall query traffic was reduced dramatically
• What traffic that did make it past memcache was well distributed onto 2 read-only slaves (db04, db04-2)
• App code was optimized to reduce overhead and unnecessary database traffic — this was done by placing hard limit on the number of search results returned by the API, among other things

How will we scale?

So these growing pains will help us move forward.  Here is our plan of attack for scaling this beast for the Firefox 3 onslaught:

• Move the API (services.addons.mozilla.org) to a separate docroot with its own read-only slaves and more aggressive caching policies that are separate from the main AMO
• Optimize client code to reduce the number of requests needed to retrieve data and also imploring local caching methods for redundant content or content that doesn’t change over time very much
• Offload even more traffic onto read-only slaves
• Upgrade to CakePHP to latest 1.1.x stable branch, which optimizes auto-generated queries quite a bit (thanks to clouserw for researching this)
• Refactor how we pull localized strings from our database
• Optimize our search performance on AMO and the API
• Switch default CakePHP data source to read-only slaves
• Find ways to use memcache at higher levels (caching larger objects instead of at just query level)

Once again it was a great team effort to get things running smoothly.  Thanks to IT for helping us troubleshoot this.  We’ll continue to build on this experience to ensure better reliability in future releases.

Looking back at the last three days, the Firefox 3 Beta 3 release was a success in more ways than one.  It showed everyone what the web can do, but it also helped us wrap our heads around the API and how much traffic it generates.  All of this will make for a better Firefox 3.0 release.

• Check it out at preview.addons.mozilla.org
• Tell us what you think or File a bug
• Check the webdev blog for updates
• Join us in IRC: #amo at irc.mozilla.org

Here are some screenshots:

Aside from a new look, here are few highlights in AMO 3.2:

• Improved developer’s control panel with statistics (see details)
• Better sandbox integration
• Simpler categories and navigation

What we’d like to know:

• Does the reskin help you find what you need quicker?
• Does the absence of “types” confuse things? (plugins, search plugins, themes, extensions)
• What should we do to make things better/easier to use for you?

Keep in mind that we are still ironing out some wrinkles. For more information:

• View in-progress 3.2 tasks
• View reported 3.2 issues
• Read the requirements and specifications
• See the original mocks

Thanks, and looking forward to hearing from everyone.

• Update sk locale from bug 367271
• Improving install experience for non-browser apps (bug 401272, r=clouserw)
• adding GUID to categories RSS feed to enable feed readers to distinguish fresh items from old ones (bug 411834)
• merging new strings from Thunderbird install experience fix (r9576, bug 401272) into all other locales
• fixing “all versions” RSS feed, bug 392183
• Fix bug 394590
• Fix bug 378782
• Total download counting in maintenance script; bug 409341; r=morgamic
• Adding pt_PT locale from bug 391197
• Update pt-BR locale from bug 380221
• Update zh-CN locale from bug 407472
• fixing data sanitization for UTF-8 characters: bug 412580, r=laura
• Adding support for application wildcards in categories; bug 408525; r=clouserw
• adding test for UTF-8 sanitization (bug 412580)
• fixing pagination sanitization, bug 412580, r=fligtar
• Checking in reviewcount column and maint script from bug 408680.
• Firefox 3 additem notices; bug 406898; r=morgamic
• fix bug 415085
• Fixing sanitization of discussion dates on addons detail page (bug 414541)
• Unflag sr-flagged add-on; bug 371214; r=fwenzel
• minor change to bin database class; bug 409341; r=morgamic
• Checking in review count column stuff from 408680. r=fwenzel.
• fixing memcaching for select queries that start with whitespace (bug 416403, r=morgamic)
• Update fr locale from bug 366239

I want to thank everyone on the AMO team for their hard work, especially localizers who have worked really hard to port AMO to their native language. 2008 is already turning out to be a great year — let’s keep it up!