Archive for March, 2007

It’s Aliiiiive!

Monday, March 26th, 2007

The new addons.mozilla.org (AMO) is up and running through its first peak load time without major issues. We will continue to monitor site performance today to verify that our performance adjustments indeed solved our load issues.

What did we do?

  • Moved public files back to releases.mozilla.org, so those mirrors can do what they are good at — delivering files.
  • Fixed image paths so they do not vary with locale. Since images are not localized, they do not have to have varying URLs. By limiting images to a unique URL, it improves cache rates.
  • Adjusted caching rules. We had configured the load balancer to not cache all content when a user has logged in. Unfortunately that meant not caching images and other high-traffic services. We adjusted caching rules to force all images and service URLs to be cached regardless of whether or not a user is logged in.

Below are the load and traffic graphs as of 1040AM PST.

mrapp01 load

The load is much more linear, and at a lower level with our adjustments in place.

mrapp01 traffic

We have an increase in traffic, but this is expected as we have many more images and locales to support. The traffic spikes from last Friday were much higher than this, however, due to releases.mozilla.org traffic and lack of caching.

Overall I can’t say how happy I am to see some better results. Thanks to everyone who worked over the weekend with us to help troubleshoot. The battle isn’t over yet — we’re still looking for ways to tweak performance — but we can definitely see the light at the end of the tunnel.

all pre-existing add-ons restored (made public)

Monday, March 26th, 2007

Short version: All add-ons that were public in AMO prior to Remora’s launch this weekend should now be publicly available on AMO again. We will be in contact with all add-on authors to provide more detail and clarity on what we’ll be doing going forward to help make the AMO and add-ons experience more polished and accessible for all our users.

Not quite as short version: I made a mistake in choosing how to roll out the sandbox and increased quality standards for AMO, in concert with the Remora software update, and we’re now correcting it. We still believe that the combination of the sandbox system (to provide more transparency and participation in evaluating add-ons before they’re published) and a higher standard for inclusion on the site (especially in terms of polish and author responsiveness) will benefit users and developers alike. But it’s clear from community feedback — often, but not always, quite reasoned and polite — that we didn’t do a good job of communicating what was going on, and as a result both users and developers were hurt by the change.

We’ve now updated the database to make all the pre-existing add-ons public, and we’ll be working with editors and users and developers to make the global improvements that we seek. The new tools and capabilities in Remora should make it much easier to serve the growing add-ons community, and we’re looking forward to even more such improvements in the future.

Thank you to everyone for your feedback, and passion, about AMO and how it affects you. Please know that we share it, and that we want to work with you to make AMO even better.

(In related news, the site performance has been improved dramatically over the weekend, but I’ll let morgamic tell the rest of that story!)

Reviewing, the Sandbox, and “missing” add-ons

Friday, March 23rd, 2007

Though Justin’s first post about the sandbox was several months ago, there is still a fair bit of confusion about the whats and wherefores of the new system. I’ll try to give a relatively concise description of the goals and current operation of the sandbox system here, but please keep in mind that we are rolling out the first version here, and not the last. We have lengthy and detailed discussions about the feedback provided by people even in its pre-release state, and we are grateful for the suggestions and information that people have contributed already.

The goals of the sandbox system, distilled down to the shortest version I can manage, are:

  1. Assist developers and site administrators in making sure that the users of AMO have great experiences with great add-ons.
  2. Expand the review system to allow participation from a much wider cross-section of the Mozilla community.

And the operation of the current review system is described at a shiny new policy page, including a long-long-long-awaited description of how add-ons can become recommended.

It is very, very important to us that unsuspecting users are protected from malicious or careless software when using the AMO site. The previous AMO version(s) relied on a system of roughly-sequential closed-door review by a small group of volunteers to validate the add-ons that were submitted to the site. As the number of add-ons grew, it became extremely difficult for that group of people to keep up with the incoming requests, and the number of specialized (and difficult-to-review) add-ons grew as well. When Firefox had 2 million users, an add-on that was relevant to 1% of those users could be reasonably left to wait a fairly long time for review, but such a “one percenter” add-on is now interested to almost a million people around the world!

Historically, Mozilla has been able to scale to the width of the web by allowing more members of our community to contribute, be it to testing, coding, marketing or dressing up in costumes and walking around downtown. We wanted to bring this increased scale to reviewing and testing add-ons to ensure that they were safe and well-described for our users.

Additionally, as Firefox’s appeal widens to a more mainstream, less computer-savvy audience, we came to realize that we needed to make sure that the add-ons were comprehensible and accessible to people who were not as comfortable working around installation problems or figuring out what an extension did by experimentation. The add-ons that people find through their use of AMO should be inspiring examples of what the extensible web can provide, and should make users’ lives better in every way. We know that the Mozilla community can provide these great experiences, and we want to both incent and reward add-on developers who provide them. We know that the AMO site is an extremely valuable channel for software to reach new users, and we want to use that to promote and strengthen the values that Mozilla stands for. We are raising the bar with Remora for what it takes to get in front of nearly a hundred million of the most savvy, influential and good-looking browser connoisseurs in the world, and we will continue to raise the bar over time as we feel it is in the interests of Mozilla and, most importantly, its users.

So we now have the sandbox, and as you’ve already read it’s where add-ons start their lives. We chose a threshold to start with to seed the public site, and we expect and hope that users of add-ons that they feel belong in the public side of AMO will write informative reviews and help the AMO editors find those gems that we — I, if you want to point a finger more closely — didn’t have in the public site on day 1.

Over time, I expect that we will make the sandbox more visible to new users, as we learn more about how to balance the need to protect unwitting users from add-ons that have not been tested with the desire to match more users up with “niche” add-ons or up-and-coming experiments. We’re already looking, based on just the pre-release feedback, at making direct links to non-public add-ons work with appropriate caveats and warnings. (They would still be hidden from search and browsing.)

I hope this helps to explain what we are trying to achieve with the sandbox system, how the system works, and how you can help make it work better on small or large scale. We’re still hard at work making sure that Remora is good and ready before we put it live, and we know that the first little while after launch will bring the need for refinement in how the system works, but we believe that these changes will work to strengthen the add-ons community and ecosystem, for the benefit of users and developers alike. That is, after all, what we’re all about.

AMO Site Updated, Rolled Back

Friday, March 23rd, 2007

AMO was updated on Thursday, March 22nd around 8pm. Overnight, we watched the web infrastructure to ensure that AMO could withstand peak load times, but this morning near peak time cluster load levels became too high and we were forced to rollback yet again to prevent affecting other critical applications.

Good news:

  • Our database bottleneck is now nonexistent
  • The app servers are fine during off-peak times
  • During this short period, we received as much if not more feedback than during previously announced beta window (2 weeks)

Bad news:

  • App server load is unacceptable
  • Traffic on web nodes has more than doubled as a result of absorbing releases.mozilla.org traffic

Strategy:

  • Profile application to look for new pain points (already done, no obvious culprits)
  • Move public add-ons .xpi traffic back over to releases.mozilla.org
  • Only sandbox files will be served from webheads for policy reasons
  • Remove locale strings from all image URLs to improve cache rate for images
  • Reassess and redeploy at earliest reasonable time

Because life isn’t complete without colorful graphs, here is a graph showing database CPU (load reflected this too) going down dramatically:remora db cpu usage
But the bad news is that our app nodes were angry:

mrapp07 load graph

This is largely due to a dramatic increase in overall traffic that was moved onto the cluster from releases.mozilla.org:

remora traffic

This traffic graph is what leads us to believe that offloading file transfers back onto releases.mozilla.org will give apache the breathing room it needs. We will post an update as soon as we can verify that.

Thanks again everyone for supporting us as we work through these issues. As for the negative comments, they serve as good motivation, too.

AMO Deployment bumped to March 22nd

Thursday, March 15th, 2007

Due to a delay in the pending Firefox release, we will be pushing AMO deployment to the evening of Thursday, March 22nd.

AMO Deployment on March 22nd

Wednesday, March 14th, 2007

AMO deployment has been moved to the evening of Thursday, March 22nd. The next Firefox release conflicts with our planned AMO deployment date and it is not optimal to make infra changes during a release.

We have had some excellent feedback reported, and we’ve made good use of our incubation period for the new site. Thank you for your feedback, it has helped tremendously! Here are some bugzilla queries if you’d like to browse through some of the feedback (and fixes):

We’d like the thank our reviewers and localizers for their contributions. They have helped us tremendously by working through the review queue, processing new submissions, and making sure our effort is a global one. So, special thanks to our friends below. You guys are as awesome as you are talented and beautiful!

Reviewers (reviews):

  • Olive (111)
  • Pontus Freyhult (76)
  • Frederic Wenzel (49)
  • Giorgio Maone (23)
  • Wil Clouser (22)
  • Nitallica (20)
  • Ken Saunders (17)
  • Mel Reyes (aka: mrtech) (16)
  • Michael J Morgan (15)
  • Justin Scott (12)
  • CP (11)
  • Archaeopteryx (10)
  • Igor Rashnitsov (9)
  • Sancus (6)
  • Davide Ficano (6)
  • Jon Stritar (5)
  • TwisterMc (4)
  • Anthony Hughes (4)
  • tnarik (4)
  • ASANO Naoyuki (3)
  • Andrew Williamson (2)
  • NeoNexus DeMortis (2)
  • Steve Jones (1)
  • Daniel Miller (1)
  • Georges-Etienne Legendre (1)
  • Felix Ritter (1)

Localizers (locale):

  • Lukas Petrovicky (cs)
  • Jesper Kristensen (da)
  • Frederic Wenzel (de)
  • Wil Clouser (en)
  • Nukeador (es)
  • Ibai Oihanguren (eu)
  • Pascal Chevrel (fr)
  • Francesco Lodolo (it)
  • Kohei Yoshino (ja)
  • Channy Yun (ko)
  • Marek A. Stępień (pl)
  • Alexander L. Slovesnik (ru)
  • Alexandru Szasz (ro)
  • mozilla.sk team (sk)
  • Ahmet Serkan Tirataci (tr)

Feedback ninja:

  • Wladimir Palant

Thanks to all of you for your hard work and contributions. There are probably others who helped out who are not listed, and to those of you we missed, thank you!

Review Queue Completed

Monday, March 5th, 2007

Pending submissions have all been processed. Thank you all for your submissions, and thanks to all our reviewers who helped us get our queue from more than 385 add-ons to 0 in the last few days. We’ve reached our goal, and will be working on migrating the entire add-ons database today.

We know everyone is anxious to get the developer control panel unlocked. Our schedule looks like this:

  • March 5 - Migrate the add-ons database and update preview.addons.mozilla.org
  • March 5-12 - Community testing of preview.addons.mozilla.org, ongoing bug fixing and polishing
  • March 12-15 - Final bug fixes and adjustments based on community testing and feedback
  • March 15 - Deployment of new addons.mozilla.org

You may have some questions. If you have any questions that aren’t addressed below, please leave a comment or find us in IRC. Here are some general FAQ:

  • When will the developer control panel be unlocked? The developer control panel will be available from preview.addons.mozilla.org on or shortly after March 5, pending the database migration which has already started.
  • Why is it taking so long to release the new site? We want to do what is best for our users. After our attempted release last month we realized we should slow down a bit and work with our community more to make sure the site is what everyone wants. We have already found and resolved a lot of bugs, and hope to find/resolve more to make the new site more robust.
  • What do we do between now and March 15? We would like the community, including developers, to use preview.addons.mozilla.org to update their add-ons. We encourage everyone to add translations where possible and test the developer tools. We also need everyone’s help testing the public pages including reviews and discussions.
  • How do you know that the site will work this time (on March 15)? After our last two attempts to switch over, we found two areas that were performance bottlenecks. One was a MySQL query for dynamic l10n data that was joining on a temp table, which means no use of indexing in MySQL. The other issue was the add-on model binding other db objects like previews unnecessarily, which caused cache entries to exceed their maximum allotted size due to series of preview images being pulled with normal add-on data. Both issues have been resolved and we have tested them using a new log replay script that we used to replay actual addons.mozilla.org traffic. The results were very positive and we have a much higher level of confidence.
  • Where do we file a bug about the new site? If you find any bugs while using preview.addons.mozilla.org, you may file them here.
  • What if we need to update something in the old database? If it is an emergency, please file an administration bug. In most cases, we will not update the old site and encourage you to update preview instead.

That’s it for now. Looking forward to a solid week of testing, and the launch shortly after. Thanks again for all of your help and support.