Posts Tagged ‘editors’

Burning down the Add-on Review Queues

Friday, November 13th, 2009

A long standing problem with AMO have been the add-on review queues and waiting times.

For those unaware, we have a review system that all new add-ons and add-on updates have to go through before they are available to the general public. The system is a combination of automatic code analysis and manual code and feature testing. The manual side is handled by a dedicated team of volunteers, known as AMO Editors. We’re a small group that has had to handle a massive amount of work, specially since the release of Firefox 3.5. We had spent the last year catching up with the numerous new submissions and updates, and the numbers will only grow once 3.6 is out.

Just 2 months ago, the new add-on review queue had over 600 pending add-ons, and the update queue had more than 250. Both had several months of waiting times, and add-on authors were increasingly unsatisfied, with good reason. Many trivial or important updates have had to wait a long time before seeing the public, causing users to complain and give add-ons bad reviews for the slow reaction of their authors.  That’s just unfair to add-on authors, and we needed a solution.

I joined Mozilla recently with a very clear objective: to reduce the review queues back to a manageable size. I have helped implement several solutions that would improve the state of affairs, like posting detailed weekly queue reports in the AMO forums, and temporarily hiring professional add-on developers to help with the burden. We estimate that our current volunteer team is large enough to keep up with the inflow, so now we just need to reduce queue size down to our ideal waiting expectations (2 weeks for nominations and 1 week for updates). After that I hope we will regain stability.

I’m happy to announce that our efforts have paid off. The latest queue report indicates that we’re down to 301 nominations, and 120 updates. That’s half of what we had 2 months ago! Just last month we reviewed over a thousand files, and our pace is increasing every week. About half of our pending reviews are within our ideal waiting time frames, and many of the ones that have been waiting for longer require an admin review for various reasons. Things are looking great.

There’s still lots to do in order to improve add-on author experience, and that is an ongoing effort. I’ve received quite a few suggestions on how to improve the reports, and I hope to implement those soon. If you’re a developer and you have anything else you’d like help with, don’t hesitate to contact me :)

Tags: , , , ,
Posted in developers | 7 Comments »

MozCamp Chile & Hispanic Add-on Development

Thursday, October 8th, 2009

I’ve just returned from Santiago Chile, where MozCamp Chile and JRSL Chile were held. It is the first Mozilla event I’ve attended, so it was a great learning experience. I specially enjoyed the fact that it was a Hispanic event so I got to meet many members of the hispanic Mozilla communities. The participant list should be enough proof that the participation was numerous and diverse. Kudos to the event organizers and all the communities, you did a fantastic job in bringing all of this together.

I made a presentation titled AMO, Add-on Developers and Editors (PDF version). It’s a very brief overview on who I am, what’s going on with AMO in terms of helping developers, and the editor community, with a call to action to anyone interested in joining the group. I was happy to realize that there were many developers in the audience, and some showed interest in becoming editors. For all of those interested, I’d look into the Add-on Policies docs and the Editors wiki page. There was also interest in localizing the Developer Hub content, which I think is an awesome idea. Having good documentation in our language is an important step we need to take if we want to increase developer adoption in our countries (I live in Costa Rica).

There was a lot of excitement around the idea of holding events that are more development-centric, like the Mozilla Add-on Workshops. There are already plans to hold one in Perú this year, and another one some time early next year. I’ll definitely be keeping an eye on those.

Thanks again to all of you who made MozCamp Chile possible!

Tags: , , , , , ,
Posted in developers, events | 1 Comment »

Add-on Review Process Redesign

Thursday, October 1st, 2009

Hello.

My name is Jorge Villalobos, and I’m the new (first, really) Add-ons Developer Relations Lead at Mozilla. I’ll be working on bringing the add-on developer community and Mozilla closer together. I have been an add-on developer for over 2 years, working on around a dozen add-ons during that time. I’ve worked on a few independent projects as well, Fire.fm being the most successful one, and the one I’m most proud of.

My initial focus in this role at Mozilla is to reduce the add-on review waiting times to a point where authors can have some certainty that their add-ons will be reviewed within a reasonable time frame. The current state of the queues is far from ideal, with the recent release of Firefox 3.5 being a big contributor to the rising tide of submissions. The queues are long, and add-on authors are not happy. I actually have a somewhat important update for Fire.fm waiting in the update queue, and I can’t help but feel a bit impatient about it.

To solve the queue situation, we are working on several solutions. We’re constantly looking for and introducing new editors to our team. We are working more closely with them to understand how they work and what their concerns are, and also to focus their efforts in the areas that have the greatest needs. We are attacking the queue problem from several different angles, some which will help us in the short term, and some which are more forward-looking, such as the one I’m introducing here.

We want to change how we handle add-on reviews, specially for updates. Our current system doesn’t handle well the fact that there are add-on authors that no longer need to have the constant scrutiny of the editor team, and don’t need to have their updates reviewed every single time. We think we need to introduce a trust factor into the process, that allows us to give more freedom of publication to authors that have proven themselves trustworthy. There are plenty of those, and I bet they are the most active authors on AMO. Reducing the amount of update reviews we give to trusted authors will give more time to our editors to focus on new add-on nominations and other updates, significantly reducing waiting times and making everybody happy.

I also cover some ideas for reviewing add-ons that are not extensions, which usually have longer waiting times when in reality they should be the easiest to check.

If you’re interested in the details, please read the proposal on Google Docs: Review Process Redesign Proposal. It’s very short, so it shouldn’t take more than a couple of minutes to read. You can take part in the discussion of the proposal in the mozilla.dev.amo newsgroup, or post a comment here. I’ll try to respond to all as time permits.

Tags: , , , , , ,
Posted in developers, policy, releases | 12 Comments »

A very busy month for the AMO team

Monday, March 16th, 2009

This last month, we’ve rolled out lots of bug fixes to AMO as well as quite a few enhancements to the site. For the comprehensive list, check out bugzilla. We’d also like to highlight some changes that we think will help both developers and users of add-ons.

Enhanced Developer Dashboards

Updated aggregations on dashboard

Add-on developers can now see more about how their add-ons are doing. On the Downloads and Active Daily Users views, you can now aggregate by week or month. You can also compare weekdays for any two weeks of data.  While we were in there improving things, Active Daily Users were also improved to count daily instead of just once a week.

Dashboard tables

In addition to improved graphs, we made it easier to view tabular data in the dashboard.  Using the very latest in HTML table technology, we present a table with the data in the aggregation you’ve selected under the summary area of the dashboard.  You’ll no longer have to export to a spreadsheet if you just want a quick view of your add-on’s performance.  We’ve also added a link to download the CSV right above the table- it’s functionality we’ve always had but it wasn’t as easy to find as we wanted.

Permalink download

Permalink Download

For the dashboards to be accurate, all your external download links should point to AMO.  In the past, this meant updating your links to point to a specific file version, as download links changed with every revision.  We’ve updated our “Add to Firefox” and “Download Now” buttons to link to a special permalink that will always serve the latest public version of an add-on.  Simple grab the URL from this link for use in your external referrals to AMO.  Please note: if you have a EULA set for your add-on, this permalink exists on the post EULA screen.

Link Sharing

Link Sharing

Sometimes there’s an add-on that you really love and want to share with friends, other times you just want to save your favorite add-on to a bookmarking service so you can keep it handy.  For these times, you can use our new Link Sharing feature.  You can use it to post add-ons to a variety of bookmarking/buzz sites.  If there’s any service we missed, let us know- we’re planning a few enhancements for this feature and always welcome feedback.

Editorial Subscriptions

For those of you nominating and updating add-ons in AMO, one of the frustrations we’ve heard is that the length of time for a re-review after a rejection is too long.  We’ve done some thinking on how to solve this, and one of the things we found was that there was no notification mechanism in place to let editors know when an add-on they’ve rejected has been updated.  Starting now, our editors will have the ability to get automatically notified when an add-on they’ve reviewed gets updated.  We hope that this will help minimize the frustration of getting an add-on reviewed and rejected for a problem that can be fixed quickly and easily.

What’s next?

As we discussed in our roadmap, we’re thinking hard about how we can make the experience better for add-on users and developers.  We’re focusing on initiatives that will help with discovery and support of add-ons, as well as enhance the developer experience by improving the workflow, analytics, and community around add-on development.  Coming up this next quarter, you’ll see changes to the end user experience that will make both featured and non-featured add-ons easier to find and discover.  Features like tagging to help users navigate to the next add-on they want, and Collections, which will make it possible for users to share and track lists of their favorite add-ons.

The sandbox process will continue to evolve, and the next step is building tools to speed up add-on review and help catch likely rejection causes before an add-on is submitted.  Finally, we’ll be opening experimental add-ons for download by logged-out users, thus making it easier for new developers to get reviews and feedback on their add-ons before nominating them for public status.

As always, we’re eager to hear what you think and very excited about the future of AMO.

Tags: , , , ,
Posted in developers, end users, releases | 3 Comments »

The How’s & Why’s of the AMO Recommended Rotation

Tuesday, March 10th, 2009

What are the Recommended Lists?

The Recommended lists are an important part of exposing AMO visitors to useful and compelling add-ons within a small & focused list. It allows us to feature add-ons that have done a good job of creating a unique and/or exciting enhancement to Mozilla software and increasing awareness of the nearly 8,000 add-ons hosted on AMO. The Recommended lists are broken down into two categories; Recommended and Category Recommended. The former is shown on the home page of AMO and is typically limited to 40 featured add-ons. The latter are lists of add-ons that are recommended at the category level. The only distinction between the two lists is that Category Recommended add-ons are not featured on the home page. Apart from that, both lists are meant to recognize the achievements of individual add-on authors and the work they’ve produced.

Every month, we perform a rotation of the AMO Recommended lists. The reason that we do this is to allow visitors to discover new & exciting add-ons to customize their software with. AMO consumers look for freshness and having the same add-ons featured monthly can make things a bit stale. Unfortunately, the monthly rotation and the methods used in determining which add-ons will be included has been a point of concern for many add-on authors. The lists are a great method to acquire more users and as such, authors work hard to get added to, and stay on, the lists. Being removed for any reason is usually a cause of concern and hopefully, this blog post will help to address the rationale behind our decisions.

The Methods Behind the Rotation

The rotation is primarily based on statistics we’ve collected to determine if an add-on:

  1. Is performing well and should be considered for Recommended or Category Recommended status
  2. Is benefiting from the Recommended lists once they’re added

The barometer that we’ve used to date to determine if an add-on should be considered for recommended status is when it’s reached at least 15-20k active users. This ensures that an add-on has gained traction, is considered popular by the community and also demonstrates that an add-on author is serious about his/her work. At times, we’ve been flexible and extended add-ons which were under that metric an opportunity to be featured especially if they were extremely unique or buried in a very busy category. On other occasions, we’ve received recommendations from someone and after finding that the add-on was indeed useful, we’ve added them on.

The methods used to generate these stats are simple. We look at pings (active users), downloads, & reviews. Before adding any add-on to the recommended lists, we physically go to their profiles and look at the reviews to see if the add-on has a number of negative reviews and if so, has the author at least attempted to resolve it. User reviews help us to determine if:

  • Users are finding the experience rewarding
  • The author is actively responding to any user concerns

Having lots of negatives reviews will count against a add-on and could lead to not being considered for recommended status or removal from the recommended lists.

Once an add-on has been added to the recommended lists, we look at the add-on’s monthly stats to determine how well it’s performing. Typically, a add-on added to the lists will see an increase in user interest and should theoretically see an increase in both downloads and active users.The stats will tell us if a add-on has increased in both downloads and active users during the previous month that they were on the list. If they’ve benefited slightly, we generally keep them on the recommended lists to see if an additional month will help them increase their users. If they’ve not benefited (eg: a drop in user retention or lack of downloads) and they were listed as Recommended (featured & on the main page), we will either drop them to category recommended or, if the decline in metrics is substantial, remove them from the lists entirely. Again, stats + reviews is what we use to determine a add-ons performance.

The point is that if a add-on is placed on the recommended lists and is not benefiting from the increased exposure, it does not make sense to continue to list them on there. Add-ons that are on the recommended lists typically experience a substantial gain in both downloads and active users so if an add-on is not demonstrating growth in any substantial way, that’s a very good indicator that the add-on may not be that attractive to AMO visitors.

The newest thing that we’ve been looking at is length of time on the recommended lists. The lists are meant for *ALL* developers to have a chance to be featured, not a select few who are either name brands or well-funded. It’s one of the reasons that there have been a number of add-ons who have been recommended for over 12 months being rotated out. This is a good thing as it allows more add-ons a chance to get exposure and it addresses one of the biggest complaints we heard at Add-on-Con; hobbyists don’t get any attention on AMO. In this last rotation alone, we’ve received kudos from small add-on developers thanking us for finally getting on the list. That speaks volumes.

Firefox Beta Compatibility and Recommended Status

Lastly, an EXTREMELY important requirement for being recommended is to be up-to-date with the most current beta version of Firefox. The number of add-ons that have been passed by due to not supporting the latest Firefox beta build (currently 3.1b2) is staggering. We’ve been announcing since December, 2008 that add-ons authors who wish to have their add-ons considered for recommended status must ensure that their add-ons are compatible. We’ve posted several articles about this and made resources available to easily update your add-ons.

This entire process isn’t perfect and it’s one of the reasons that we’re looking to expand the distribution channels available to developers. Getting everyone more avenues for additional exposure is a top priority for Q2 and we’re focused on improving the process.

Tags: , ,
Posted in developers, policy | 6 Comments »

Successfully Getting your Addon Reviewed

Wednesday, January 14th, 2009

Thanks to AMO Editor Cesar Oliveira for allowing us to repost this here:

As the addon review queue grows beyond 600 nominated and updated addons, it is more important than ever to make sure your extension passes review the first time around. Editors have a review guide, and a well hidden but publicly viewable set of policies. But here is a quick list of the most cited reasons for addon refusal :

  1. Addon should have at least 3 user reviews before being nominated. External reviews count too, but you have to mention them in your nomination message. Softpedia reviews do not count.
  2. Wrap your loose variables. All objects, variables, and anything that your addon makes global should be wrapped to avoid conflicts with other addons.
  3. Look at the error console. Is your extension throwing up javascript errors? Fix it. There are some options you can set in about:config to help you with this.
  4. You break functionality in the host application (Firefox, Thunderbird, etc.).
  5. Your extension doesn’t work properly, or showing unexpected results. This is what user reviews are expected to catch, but you really should get someone who wasn’t involved to test and report bugs.

There are other less common things that can get you busted too. I’ll just list them here for completeness sake :

  1. Including remote javascript/css or other files, anywhere. Include them in your xpi file instead.
  2. CSS warnings on the error console if your submitting a theme.

If you have any doubts, check the public policies page linked above.

And don’t ask when your extension will be reviewed in the comments. Because every answer will be the same : When someone gets around to it. It could be sooner if you’d help!

Tags: , ,
Posted in developers, guest posts | 3 Comments »

AMO Roadmap & Directions, Firefox Summit Style

Monday, August 4th, 2008

At the recent Firefox Plus Summit, I missed the opportunity to share my slides for the AMO session due to a power outage. Ended up using lots of hand gestures and flip charts. For those interested, I’ve uploaded them here and welcome any questions or comments.

(Also, uploaded the slides from the AMO Editor’s session).

Thanks.

Tags: , ,
Posted in developers, end users | 5 Comments »

Changes to the Featured & Recommended add-ons on AMO

Thursday, April 24th, 2008

One of the main goals for AMO is to help introduce add-ons to the user base of Firefox (and other other apps supported on AMO). The recommended list has been one of the ways to do that.

Prior to AMO v3.2, a single list of add-ons was maintained that showcased a sample of what can be done with add-ons. This came to be known as the “Recommended List”. Participation in the recommended list was coveted by authors since it was one of the more visited pages on the site and meant your add-on got downloaded more. AMO currently receives about 5 million pageviews per day with approximately 3% of the traffic to the recommended list. The list has grown and changed over time but has had about 25 add-ons on it. With a single list, the members of the recommended list got the most attention and “starved out” visibility for other add-ons.

One of the changes that was introduced with the latest AMO redesign is the ability to highlight a wider variety of add-ons. Instead of a single list, recommendations are now on a per-application, per-category and a per-locale basis. This gives the community greater flexibility and  increases the face time and exposure (impressions) that an add-on can have.

With AMO 3.2, we now have many lists.

  • Category-Recommended lists are edited lists managed by the AMO editors. They represent a list of the most interesting add-ons for a particular category.
  • Category-Recommended add-ons appear on:
    • One of the two slots in the AMO category landing page (see sample)
    • The Recommended List page for that category (see sample)
    • The default view in the “Get Add-Ons” panel in the Firefox 3 Add-Ons Manager (once this bug is fixed).
  • We anticipate about 5-10 recommended add-ons per category. (Fuzzy math: 14 categories x 10 add-on/category = 140 recommended. Approximately 5000 add-ons on AMO means 3% are recommended.)
  • As AMO editors are constantly reviewing add-ons, they can opt to showcase promising add-ons
  • Add-on authors can request to become recommended (see last question)
  • The Featured list is an edited list managed by the AMO administrators and editors that showcases add-ons for a period of time. This list will have add-ons rotate into and out of it on a periodic basis. Currently, once every 3-4 weeks. The list will be chosen from the existing set of Category-Recommended add-ons.
    • Members of the Featured list will be rotated into one of three slots on the front page of AMO
    • In order to ensure that the feature list maintains a wide sampling of add-ons from various categories, not all add-ons will be swapped out since some categories will need representation.
    • We anticipate about 20-30ish add-ons to be on the featured list at any one time
    • Currently, the following add-on categories will not be chosen for participation in the Featured list: Web Development, Language Tools & Toolbars. This may be revisited at a later date.

So, check out the new AMO site and the latest batch of Category-Recommended add-ons as well as the most recent rotation of Featured add-ons.

We’ve tried to outline a policy around how all this will be managed – if you’d like to read and review the gory details on how the featured and recommend lists work – check this out. Feedback is appreciated as we are trying to be as transparent as possible. There’s more to do in this area including clarifying some edge cases on policy, considering whether to recommend themes and building locale-specific Category-Recommended and Featured lists.

Tags: ,
Posted in developers, end users, policy | 7 Comments »