Mozilla Webdev

The download controller was modified on Thursday to prepare for the release of the 1.5.0.12 -> 2.0.0.4 major update. During release cycles, AMO takes abnormally high load which sometimes causes interruptions in service.

To avoid this situation we agreed to cache public download hits from the AMO install buttons. This does two things:

• Relieve application load by allowing the hardware load balancer to cache file requests — which are ultimately redirects to releases.mozilla.org
• Relieve the database by not having constant inserts on the download table — which causes extra load because of indexes that were put on the table to make count updates work correctly

The benefits can be seen below. First, traffic for a web node:

Second, a CPU usage graph from the same node:

You can see the positive effect on load near noon on Wed.

The status of download counts is tracked in bug 384084. The plan is to create a background maintenance script that parses AMO logs via a cron job in off-peak hours. Counts will be updated once a day.

In addition to this, we are taking the opportunity to do a couple of things. For our summer goals, we plan on improving statistics for developers by offering:

• Actual update ping counts in bug 384086
  
• An improved API for aggregating add-on statistics and integrating it into your blog or external sites

We plan on resuming download counts early next week and update pings should be available by mid July. Thanks for your patience as we make adjustments to accomodate server load, and have a great 4th of July!

Steve Krug would agree that thinking is a bad thing. Not that thinking in general is bad. I’ve found that it is actually a good thing.

Thinking can be bad when it is a barrier between a user and what they want. People don’t want to spend more time thinking about how to do something — they just want to do it.

It is why simpler sites win. Simpler means less thinking and a better experience for the average user.

The same ideology can and should be applied to all other realms. The “waste no brain cells” approach is often applied in marketing and user experience areas during the design phase of a web application. But on a larger scale, shouldn’t it be applied to an entire organization? To a community? To their actual ideas?

Take a look at Dell’s idea storm and I think you’ll get the idea (or ideas!). They are taking a fresh approach to consumer affairs and empowering users with the ability to tell them what they think — without thinking too much in the process.

The concept behind this site is simple. Users have all the good ideas, and a company ultimately wants to please its users. It is a tool to bridge the gap between Dell’s future plans and the dreams of its community. After all, shouldn’t they be the same? It’s not ALL about profit, is it?

So I say bravo, Dell, and I hope it works out. But it seems like a lot of ideas just sit there for a while. We’ll see how it goes. Ones I liked in particular:

• Have Firefox pre-installed as default browser
• IdeasStorm needs more participation from Dell Management and Employees

They lead me to wonder — who is considering items that are **UNDER CONSIDERATION**?

In open source I don’t think projects or organizations lack feedback or tools for aggregating feedback. It’s just that they are too complicated and seem daunting to someone who just want to tell us what they are thinking.

I worry about the alienation of a large percentage of users. In the Mozilla community, specifically, we rely heavily on Bugzilla as a bulletin board for user feedback. In order to report a bug and let their ideas be known, a user has to take a few steps:

1. Create a Bugzilla account
2. Sign in
3. Figure out what product they need file a bug in
4. Figure out what component they need to file the bug in
5. Figure out what all the other stuff means
6. Follow Bugzilla rules on how to file a bug
7. Realize they should look for dupes
8. Look for dupes
9. File the bug
10. Wait a while and hope something happens

Sorry, you lost John Doe on step 1. He decided to grab a beer and watch the Sopranos instead. It requires less thinking and is much more entertaining.

Reporter, hendrix and the uninstall survey also attempt to gather user feedback but once it gets there it stagnates. There’s no active participation that occurs after the delivery of an idea.

So what about this?

1. Click on something
2. Say what you think (while optionally creating an account if you want to track your idea)
3. Hit submit
4. If it’s a great idea and you’re up for it, track responses and participate in resulting discussions

I can see a need for this in almost every organization — small or large. I think the advantages would be huge and the investment relatively small given the frameworks we have at our disposal and the simplicity of the application.  What do you think?

The last time Kubla came up we were still evaluating options and the code was an old version of CakePHP all flushed out with scaffolding. Most of the time between then and now has been spent on other projects (I’m looking at you AMO) but we’ve made enough progress to justify another update.

Out of the CMS’s evaluated, we thought two showed great promise for our set of requirements.

The first option that I got excited about was Wyona’s Yulup. This is actually an add-on for Firefox that can talk directly to SVN (via WebDAV). It is simple, supports WYSIWYG editing, has xhtml validation, supports WebDAV authentication, and much more, all out of the box. I like that it leverages existing systems in an effective way, and is fairly unique (compared to traditional CMS’s). It has a lot of promise for someone looking for a simple solution to editing static pages.

That said, we would have to make significant changes to the add-on, or, more likely, write quite a bit of code separately to handle requirements like indicating a relationship between a page in English and other languages (to tell which English pages had been translated and which hadn’t).

The second option is the well known Drupal. There are lists expounding on Drupal’s good qualities already, so I’ll skip that, but suffice it to say, we’re most interested in the modularity that comes with Drupal. We’re going to need to write some code to handle language detection and localization, and Drupal’s hooks should make this easier, not to mention the pile of pre-built modules.

After collaborating on our options, we’ve decided to pursue the Drupal option. I’ll be replacing the languishing code in SVN with a copy of Drupal shortly, and continuing development there.

Thanks for everyone’s input.

Wednesday was the first Firefox release since AMO 3.0 (Remora) launched in late March. It’s expected that traffic to Mozilla websites will increase following a release, but it’s usually in the range of 1.5 times normal traffic. Thursday, traffic to our San Jose facility tripled normal traffic, breaking 600 Mb/s. (The historical graph below is averaged down and doesn’t show that high, but Justin will be giving more details from IT’s point of view soon.)

When Firefox is updated, a separate update check for each add-on installed is performed, causing AMO to get quite a bit of traffic. Thursday, however, much of the traffic AMO was seeing was not from the update check - it was from real people searching, downloading, and browsing the site.

We had over 2000 user accounts created Thursday alone, over three times a normal day. 1 out of every 17 people that saw the What’s New page after updating Firefox clicked on the “Firefox Add-ons” link. Both sessions and pageviews on addons.mozilla.org tripled on Thursday - not including update and blocklist pings. The number of add-on downloads more than tripled from 2 days before.

While all of this increased activity is great news for us, it wasn’t so great for our app cluster. We had a number of issues throughout the day ranging from memcache hitting connection limits and refusing connections, database server issues, and app servers dying in a domino effect. A huge thanks to IT for keeping the issue under control the whole day, especially mrz who was on call and did not get to sleep.

We were able to make a number of changes Thursday and quickly push them to production, such as directing search to the shadow/read-only database server, adding to the list of areas of the site we can disable if necessary, and adding more memcache servers. We’ll be evaluating what we can do to prepare for this if it happens again.