The Add-on Review Process and You

The add-on review process remains a mystery for many add-on developers. As a developer myself, I admit it feels like dropping your add-on into a bottom-less pit and just waiting (and hoping) for something to happen. As the weeks pass by, patience runs out and you wonder what’s going on.  Developers have rightly demanded more transparency in our review process, and the purpose of this post is to explain where we are now, and what we’ll doing in the future to improve it for all add-on authors.

Starting next week, I’ll be posting regular updates on the state of the queues and the review process in general. This post explains how the overall process is functioning now, which will allow me to keep the updates as concise as possible. Whether you’re a new add-on author, a veteran developer, or an add-on enthusiast,  I recommend you take the time to read this and ask any questions you think are unanswered here.

The Review Process

AMO uses a sandbox system. All add-on files uploaded to AMO are placed in the sandbox, where only a minority of users can locate and install them. They are currently labeled as “Experimental” on AMO listings, but may be re-labeled to “Unreviewed” or something similar in the future.

All uploaded files are scanned for malware (viruses, trojans, etc.) using a variety of tools. A recent security problem exposed a vulnerability in our system, and that was that we were relying on a single virus scanner which wasn’t all that good. Our scanning system is now more robust, and we’re considering other ideas on how to improve security in the sandbox system.

In order for an add-on version to become public and readily available to all, it needs to be submitted for review by its author, and pass the review process.

For new add-ons, which we usually call nominations, the review process goes as follows:

1. The author goes to the Developer Hub and creates a new add-on listing by uploading the add-on file and adding information such as descriptions and preview images.
2. The uploaded file is run through our Code Validation Tool. This tool warns the author about possible code quality issues which may be reason enough for rejection. The Validation Help page explains in detail the reasons behind the checks and which are more important to pay attention to.
3. The author can nominate the add-on as part of the submission process, or later from the add-on status page accessible through the Developer Hub. Having descriptions and at least one preview image are requirements for nomination. We no longer require a minimum amount of user reviews for nominations, but having some may improve chances of approval.
4. After nomination, the add-on status page will indicate a status of “In Sandbox; Public Nomination”. This means the add-on is in the nomination review queue.
5. When the add-on is reviewed, it will either be approved, denied, or additional information may be requested from the author. An email sent to the author will contain in detail what the reviewer though about the quality of the add-on and possible areas for improvement. In case of rejections, these notes should all be taken care of before nominating the add-on again. The approval message may also contain suggestions, and even requirements for the next release, so authors should always read the message entirely.

Approved add-ons will have a new “Public” status; denied add-ons will have an “In Sandbox” status. Authors can reply to the email in case they want to discuss the notes directly with the reviewers. Thank you notes are also appreciated :).

After an add-on becomes public, new versions of the add-on (updates) undergo a similar process:

1. The author goes to the Developer Hub and creates a new version of the add-on by uploading the add-on file and adding version information.
2. The uploaded file is run through our Code Validation Tool. This tool warns the author about possible code quality issues which may be reason enough for rejection. The Validation Help page explains in detail the reasons behind the checks and which are more important to pay attention to.
3. The new version will be automatically nominated for the public (except for some bugs). The status of the latest version can be easily verified in the My Addons page, in the Versions and Files section. It should be “In Sandbox; Public Nomination” at this point. If it isn’t, let us know.
4. When the update is reviewed, it will either be approved, denied, or additional information may be requested from the author. The email sent to the author will contain in detail what the reviewer though about the quality of the add-on and possible areas for improvement. In case of rejections, these notes should all be taken care of before updating the add-on again. An email sent to the author will contain in detail what the reviewer though about the quality of the update and possible areas for improvement. In case of rejections, these notes should all be taken care of before submitting a new version. The approval message may also contain suggestions, and even requirements for the next release, so authors should always read the message entirely.

Approved versions will have a new “Public” status and will be automatically pushed to users that have the add-on installed. Denied versions will have an “In Sandbox” status. Authors can reply to the email in case they want to discuss the notes directly with the reviewers.

At any point in this process the author can send a message to amo-editors AT mozilla DOT org to request information about their add-on review status.

The Review Queues

A common concern for add-on authors is “How much time will I have to wait for my nomination / update to be reviewed?”. This is difficult question to respond, given how different add-ons are and how the reviewing team works.

As explained before, all add-ons nominations and updates are placed in a review queue. There’s a queue for nominations and a separate queue for updates. They are both sorted by waiting time, the oldest pending reviews being at the top. They are reviewed by a team of volunteers (mostly), who give higher priority to oldest reviews. Reviews have wildly varying levels of difficulty, though, and that’s the reason that a fraction of them take specially long to be done. The most complex ones are reviewed by a senior or admin editor, usually me. If I reviewed your add-on in the past couple of months, chances are that it is much more difficult to review than the norm.

I post queue status reports every Friday in the Add-ons Forum. In these reports you can see a breakdown of each queue by waiting time ranges, and showing the progress we’ve done in the previous 2 months. Our current goals should be clear from the reports: all updates should be reviewed within a week, and all nominations should be reviewed within 2 weeks. The latest report shows we have pretty much achieved this goal for updates, and we still have work to do for nominations. Compare these numbers with the very first report I posted last September, add to this the fact that Firefox 3.6 was released recently (lots of new add-ons and updates), and you’ll realize just how much things have changed with the queues in the past 5 months.

As a little bit of trivia, in September there were approximately the same amount of nominations waiting for longer than 4 weeks (305!) than the total amount of add-ons we had last Friday in both queues combined (311). Today, updates are normally reviewed within a couple of days, and nominations are normally reviewed within 3 weeks. And all of this is improving every day, thanks to…

The AMO Editors

Add-on reviews are performed by the AMO Editors team. Everything you ought to know about us should be accessible from that link. If there’s more you would like to know, please contact us and we’ll update the wiki accordingly.

From the wiki:

AMO Editors is a Mozilla community dedicated to guard the security and reliability of add-ons listed on AMO. As part of the add-on review process, editors review the code and functionality of new add-ons and add-on updates, and decide if they’re appropriate for publication or not. Editors follow and enforce the established AMO Policies.

Most editors are volunteers; experienced extension developers that want to participate in the review process and improve add-on quality and security on AMO. Some of us are Mozilla employees, and a few others have been contracted to help us in our effort to bring sanity back to our queue lengths. We have 74 registered editors, but only a little over a dozen are active at any given time. We receive many applications for new editors, and a new member is added to the team almost every week.

There’s some information in the AMO Editors wiki page that authors will find very valuable:

• Useful information for Editors. This page explains in more detail how we perform our reviews, and some additional reviewing guidelines that may not be specified in the official policy documents. This page and the pages it links to are the basis of our editor training process.
• Useful information for Add-on Authors. How to tell if your add-on is in the queue, tips on how to make sure your add-on is reviewed quickly and many useful reference links.
• How to Become an Editor. Our doors are open and we appreciate your help :).

Most of this information was added to the wiki fairly recently. I’ve been trying to figure out how to better expose it on AMO, and I think this blog post is a good start. There are a few places where I think it will be useful to link to this blog post.

The Future

For the sake of transparency, authors are requesting us to open up the review queue and make it fully visible to the public. As you’ll see in my comments on that bug, I don’t think that’s such a good idea. As I explained before, add-ons are not reviewed strictly in order. Some may remain in the queue for longer times while others are reviewed instantly, depending on several factors. We already receive plenty of complaints about this in the mailing list, and I think that opening up the queue will multiply these complaints significantly, specially considering that some add-on authors are very competitive (sometimes businesses dedicated to their add-ons). Since we have to manage the mailing list as well, I think our work will be frequently interrupted by “why X and not me” messages if we chose to do this.

However, there are a few things that I think we can do right now that will help improve the queue visibility situation. I’ve opened them up as separate bugs:

• Promote more information about the review queues on AMO.
• Notify Authors when their add-on has been flagged for Admin Review. This is one of the main causes for review delays, and right now we don’t notify authors about it.
• Add-on status page should show add-on place in the review queue. Knowing where your add-on is in the review queue would be a great improvement and it wouldn’t require to disclose the full contents of the queue.

Ideas are welcome of course. Please comment on the bugs that are important to you. Open new ones if you think there’s something else that needs fixing. Comment on this bug, or send me a message to jorge AT mozilla DOT com. Above all, let’s keep the communication as open as possible.

Oh, and congratulations if you made it all the way to the end of this post 😛