Burning down the Add-on Review Queues

A long standing problem with AMO have been the add-on review queues and waiting times.

For those unaware, we have a review system that all new add-ons and add-on updates have to go through before they are available to the general public. The system is a combination of automatic code analysis and manual code and feature testing. The manual side is handled by a dedicated team of volunteers, known as AMO Editors. We’re a small group that has had to handle a massive amount of work, specially since the release of Firefox 3.5. We had spent the last year catching up with the numerous new submissions and updates, and the numbers will only grow once 3.6 is out.

Just 2 months ago, the new add-on review queue had over 600 pending add-ons, and the update queue had more than 250. Both had several months of waiting times, and add-on authors were increasingly unsatisfied, with good reason. Many trivial or important updates have had to wait a long time before seeing the public, causing users to complain and give add-ons bad reviews for the slow reaction of their authors.  That’s just unfair to add-on authors, and we needed a solution.

I joined Mozilla recently with a very clear objective: to reduce the review queues back to a manageable size. I have helped implement several solutions that would improve the state of affairs, like posting detailed weekly queue reports in the AMO forums, and temporarily hiring professional add-on developers to help with the burden. We estimate that our current volunteer team is large enough to keep up with the inflow, so now we just need to reduce queue size down to our ideal waiting expectations (2 weeks for nominations and 1 week for updates). After that I hope we will regain stability.

I’m happy to announce that our efforts have paid off. The latest queue report indicates that we’re down to 301 nominations, and 120 updates. That’s half of what we had 2 months ago! Just last month we reviewed over a thousand files, and our pace is increasing every week. About half of our pending reviews are within our ideal waiting time frames, and many of the ones that have been waiting for longer require an admin review for various reasons. Things are looking great.

There’s still lots to do in order to improve add-on author experience, and that is an ongoing effort. I’ve received quite a few suggestions on how to improve the reports, and I hope to implement those soon. If you’re a developer and you have anything else you’d like help with, don’t hesitate to contact me :)

Tags: , , , ,

7 Responses to “Burning down the Add-on Review Queues”

  1. I know there were ~ 470 add-on nominations when I nominated mine and I thought it would take 3 months or more; mine was reviewed in under a month. Outstanding work! This is going to make a significant difference to the overall add-on experience.

  2. Alex says:

    Comment has nothing to do about your latest entry but I thought I heard that Thunderbird was suppose to e updated this past week. What happened or war this rumour just bull?

  3. Alfred Kayser says:

    It is indeed improving, but still a large backlog to work away, and I really appreciate the hard work here.
    My Nautipolis for Seamonkey is now 5 weeks in review (posted October 6th). Considering that this is a theme (which requires less strict checking), and that it is a old friend of Mozilla (since 10 years now) and Firefox (since 5 years) it should not to be too hard to get it reviewed…

  4. Mark Smith says:

    Thanks for the informative post. There is a lot of frustration on the part of add-on developers and publishers. Is there a way to tell if an extension update is in the review queue?

    For example, I submitted an update for Pearl Crescent Page Saver Basic 6 weeks ago and it has not been approved.

  5. Jorge says:

    @Mark Smith: in the developer tools, you should see a Files and Versions section, where you can see the current status for all versions of your add-on. If your last version has a status of “Sandbox, Nominated for the Public”, then it means it’s on the queue. If it only says “Sandbox”, then you didn’t nominate it when you uploaded it, or it was rejected (but you should have received an email in that case).

  6. Brian King says:

    Great work Jorge!

  7. Alfred Kayser says:

    Hi Brian, thanks for approving and nominating my themes!
    AMO is just great!

Leave a Reply