Archive for the 'AMO' Category

The Curious Case of the Giant Scrollbar

Tuesday, February 3rd, 2009

Recently I fixed bug 439269 (”AMO theme has unnecessary scrollbar at the bottom”) and thought it was an interesting bug for a few reasons.

To summarize the issue, for no apparent reason in right-to-left languages a really long scrollbar would appear at the bottom of the window.

Screenshot of scrollbar

Even though there was a scrollbar, when you scrolled all the way to the left, nothing was there. Another reason this was odd was the scrollbar only appeared in right-to-left (RTL) languages. Inspecting the page via Firebug didn’t give any clues as to what was causing the issue as there was no element hidden somewhere onscreen. Finally, to make things even weirder, the scrollbar only appeared when JavaScript was turned on.

After some thought, I had a feeling that we might be using absolute positioning to position an element to the left and above the page offscreen, which is quite common. In a RTL page, however, left is does not move an element outside a page’s boundaries. So the result is you get a scrollbar.

So what’s a web developer to do? Firebug to the rescue! I popped it open and started typing some JavaScript into the console to find an element that seemed really far offscreen:

var nodes = document.getElementsByTagName("*");

for(var i=0; i < nodes.length;i++) {
    var node = nodes[i];
    if(node.offsetLeft < -500) {
        console.log(node);
    }
}

And Firebug’s console spit out:

<ul id="cat-list">

Ah-ha! Now I was getting somewhere. A quick search through our CSS files for ‘#cat-list’ found an interesting line of code:

#categories.collapsed #cat-list {
    position: absolute;
    left: -999em;
    top: -999em;
}

And when JavaScript is turned on, the class ‘collapsed’ is added to the parent node #categories. In RTL mode this creates a huge scrollbar because -999em to the left of the page is a valid location that a user can scroll to. The solution?

.html-rtl #categories.collapsed #cat-list {
    position: absolute;
    left: 999em;
    top: -999em;
}

On any pages that are RTL, we add the class ‘html-rtl’ to the body tag in order to change the layout for RTL languages. This fixes the issue by moving the category list offscreen to the right, which is outside the page in RTL mode.

Things to remember:

  • Firebug is your friend
  • The DOM is a live document you can inspect, utilize this feature
  • Be careful with positioning with sites that are LTR and RTL

AMO API Fennec Support

Friday, October 31st, 2008

The AMO API supports Fennec now. We pushed Bug 453517 to update services.addons.mozilla.org to work with the Fennec alpha and future Fennec versions.

AMO Statistics and CDN Update

Monday, September 15th, 2008

We will be adjusting log collection to work with our CDN tonight during a maintenance window. We can’t guarantee that developer statistics (download counts and update pings) will be 100% continuous — so please comment here or file a bug if you see anything strange.

Thanks for your help!

AMO 3.4.3 delayed until June 12th

Friday, June 6th, 2008

We attempted to push the AMO 3.4.3 update last night but ran into some problems. Our initial investigation is pointing to the advanced search queries taking far longer than the previous version. The slave databases couldn’t handle the load and we had to back the update out. We’ll come up with a solution this weekend and plan on pushing the 3.4.3 update on June 12th.

AMO 3.4.2 update scheduled for tonight (Thursday)

Thursday, May 15th, 2008

As previously mentioned we’re planning on updating addons.mozilla.org for the 3.4.2 changes tonight. There are 32 bugs that will be fixed with the update.

Please let us know if you see any regressions or anything you’d like to see changed.

The next update is scheduled for May 29th and will be mainly a bug fix release with minor new features (like bug 432669). There are currently 28 bugs scheduled for the 3.4.3 push.

AMO 3.4.1 update scheduled for Thursday

Tuesday, April 29th, 2008

The AMO team is finalizing the latest update, 3.4.1. This is a bug-fix release addressing things like double escaping, improper L10n redirects, and category cleanup. A list of all the bugs targeted for this release is available.

We’ll be committing the last of the patches very shortly and the changes will be available on preview.addons.mozilla.org. Feel free to look at the changes and send us any feedback you have. We expect to push the changes live this Thursday evening (May 1st).

Our next release, 3.4.2, is currently scheduled for May 15th. 3.4.2 will also be a bug-fix release - there are currently 19 candidates.

AMO 3.2.1

Monday, April 21st, 2008

addons.mozilla.org was updated last week. AMO 3.2.1 was a maintenance release (26 bugs fixed) for any major issues with 3.2.

Our next release will be AMO 3.4.1, the first of three dot releases for AMO 3.4, which is our next milestone to be completed before Firefox 3.

AMO 3.2 SVN Stats

Thursday, March 27th, 2008

Ran statsvn on r7797 through r11622.  View svn stats for AMO 3.2.

All week, all the time:

Activity per day of the week

On the developers page, you will see how many non-mozilla contributors have helped us update translations.  Overall, some rough stats:

  • 3134 changes
  • 288602 lines affected
  • 24 contributors

Again, thanks a ton for your help!

Bringing Sexy Back to AMO

Wednesday, March 26th, 2008

Yep — “sexy” and “AMO” in the same sentence.  Our festively plump 3.2 target milestone added many features and refreshes that you can read about in a great post on Basil’s Bodacious blog.

In the same time period we’ve also:

  • Upgraded CakePHP’s core software from 1.18.x to 1.19.x
  • Added support for weighted database slaves
  • Migrated AMO to PHP5 off of the now end-of-life’d PHP4
  • Replaced Scriptaculous/Prototype with jQuery
  • Improved IE6/IE7 compatibility
  • Improved accessibility features

And really, when it comes to sexy, the real magic starts with our volunteers.  Our editors have worked hard to review new and updated add-ons as we move towards Firefox 3 this year and our localizers translated roughly 200 new strings in AMO templates in a little over three weeks for 24 locales (wow).

Thanks to everyone for pitching in to make this release happen.

Hungry Hungry Add-ons Manager

Friday, February 15th, 2008

For about 48 hours the AMO API was thrashing because of the popularity (and hunger!) of the new add-ons manager in Firefox 3 Beta 3.  The dust has settled and the servers are humming happily along, so now is a good time to blog about what happened and how we’ll handle future releases successfully.

Stop.  Take a deep breath.  Alright, here we go.

What happened this week?

Now that the API is functional (most major bugs have been ironed out) we got a rude awakening this week and found out exactly how much traffic the improved Add-ons Manager can generate, but it’s a nice problem to have and we’re happy it’s been well received.

Wednesday, around peak time, the API started clobbering our databases:

db03 load average

Shortly after we entered our peak traffic window, we had to turn off the API to keep the normal AMO working.  Diagnosis found that:

  • Load was not utilizing the read-only slave and was focused mainly on the master read/write database (mrdb03).
  • Cache hit rates were down to 60% from the usual 90% for memcached
  • When our databases hit peak CPU, the app cluster would tumble because of the piling requests

How it was fixed?

Wednesday, IT and Webdev spent quite a bit of time getting the API back up.  Starting with the three points above, we:

  • Off-loaded read-only traffic to DB slaves
  • Investigated optimizations for the API
  • Looked at cache rules and cache policies for both memcache and the hardware load balancer

However, Thursday didn’t fare any better for the cluster.  This time the slaves started to melt near peak time — forcing us to once again temporarily disable the API.  Under-utilizing memcache was the main issue.  Cache headers were fine, slave was utilized, app nodes were fine — just too many damn queries flying at our database servers! :)

Load got high, but we disabled the API before it became critical

So on Thursday we continued our look into what was going on.  We tried to figure out why our cache hit rate was so low (60% instead of 90%).  Digging through AMO, we found CACHE_PAGES_FOR, which set the expire time on memcache records when calling Memcache::set(), was set to 60 seconds.  We increased this to 7200 to aggressively cache database traffic and were collectively off for valentine’s dinner.

The next day, Memcache was our valentine.

mrdb03 survived Friday without a blip

db04 load was higher than the read/write master
The combination of our efforts worked:

  • Overall query traffic was reduced dramatically
  • What traffic that did make it past memcache was well distributed onto 2 read-only slaves (db04, db04-2)
  • App code was optimized to reduce overhead and unnecessary database traffic — this was done by placing hard limit on the number of search results returned by the API, among other things

How will we scale?

So these growing pains will help us move forward.  Here is our plan of attack for scaling this beast for the Firefox 3 onslaught:

  • Move the API (services.addons.mozilla.org) to a separate docroot with its own read-only slaves and more aggressive caching policies that are separate from the main AMO
  • Optimize client code to reduce the number of requests needed to retrieve data and also imploring local caching methods for redundant content or content that doesn’t change over time very much
  • Offload even more traffic onto read-only slaves
  • Upgrade to CakePHP to latest 1.1.x stable branch, which optimizes auto-generated queries quite a bit (thanks to clouserw for researching this)
  • Refactor how we pull localized strings from our database
  • Optimize our search performance on AMO and the API
  • Switch default CakePHP data source to read-only slaves
  • Find ways to use memcache at higher levels (caching larger objects instead of at just query level)

Once again it was a great team effort to get things running smoothly.  Thanks to IT for helping us troubleshoot this.  We’ll continue to build on this experience to ensure better reliability in future releases.

Looking back at the last three days, the Firefox 3 Beta 3 release was a success in more ways than one.  It showed everyone what the web can do, but it also helped us wrap our heads around the API and how much traffic it generates.  All of this will make for a better Firefox 3.0 release.