Mozilla Webdev

Over the last week, we’ve encountered some problems in our monitor and processor caused by a large number of pending jobs:

• main monitor thread takes > 1 hr to complete a full scan of pending jobs on disk
• priority job processing depends on this thread

In order to fix these delays, we have made priority job monitoring a separate thread from the main queue thread, but we are working to reduce the delay back to 1-2 minutes (which is what we’re used to!).

Currently, we are blocked by issues with this new method related to filesystem scanning.  This is blocking the archiving and data re-import mentioned in our last set up updates.  See the related code.

We are working to fix this asap, and will provide updates this evening.

The Socorro database, which is the main database for Firefox 3 crashes, will be going through some maintenance upgrades this week. Starting tonight, this means:

• Report data prior to June 23rd will be temporarily unavailable.
• New reports will work as usual on our new database.
• Old data will be imported into the new database using an improved partitioning plan.

Reasons why we need to do this now:

• Firefox 3 crash throughput has been about triple the projected amount (due to the popularity of Firefox 3).
• The current database has a partition that is unmanageable without significant downtime (2-3 days)
• The Socorro reporter is not responsive because of the size of the current partition.

No individual reports will be lost, they will be restored over the next week. Please let us know if you have any questions.

Les came up with a way to syndicate the Download Day JSON feed. You might want to check it out.

Last week we had our entire team on-site and had discussions about where we’ve been and where we’re headed. I put together a short presentation about past, present and future then we had a short brainstorm session about “ok, what’s next?”.

We wanted to share all of this with you to invite any additional feedback or ideas for how we can make the Mozilla web better. Below are the notes from our brainstorm session, and I invite you to add to the wiki page or comment here.

Our main questions for the future that we’d like your thoughts on are:

• What would you like to change about our websites?
• How do we encourage more community participation in development?
• Give us a project idea!

Again, check out our brainstorm notes and we’d like to hear from you if you have any ideas to add to this list! Thanks.

We attempted to push the AMO 3.4.3 update last night but ran into some problems. Our initial investigation is pointing to the advanced search queries taking far longer than the previous version. The slave databases couldn’t handle the load and we had to back the update out. We’ll come up with a solution this weekend and plan on pushing the 3.4.3 update on June 12th.

Due to the holiday on Monday we’re going to delay the AMO 3.4.3 update this week until June 5th.

Aside from some bug and security fixes in this release we’re mainly focusing on search improvements, streamlining the editors’ queue, and making it easier to browse and discover themes.

There are currently 43 bugs targeted for the 3.4.3 update about half of which are fixed and verified. When bugs are resolved you can view the changes on preview.addons.mozilla.org. As always, feel free to send us any feedback you have.

As previously mentioned we’re planning on updating addons.mozilla.org for the 3.4.2 changes tonight. There are 32 bugs that will be fixed with the update.

Please let us know if you see any regressions or anything you’d like to see changed.

The next update is scheduled for May 29th and will be mainly a bug fix release with minor new features (like bug 432669). There are currently 28 bugs scheduled for the 3.4.3 push.

The AMO team is finalizing the latest update, 3.4.1. This is a bug-fix release addressing things like double escaping, improper L10n redirects, and category cleanup. A list of all the bugs targeted for this release is available.

We’ll be committing the last of the patches very shortly and the changes will be available on preview.addons.mozilla.org. Feel free to look at the changes and send us any feedback you have. We expect to push the changes live this Thursday evening (May 1st).

Our next release, 3.4.2, is currently scheduled for May 15th. 3.4.2 will also be a bug-fix release - there are currently 19 candidates.

Last Friday we pushed some important updates to Socorro:

• Bug 426940 - Reduce or eliminate delay in collector to monitor hand-off
• Bug 426940 - Fix processor handling of error conditions
• Bug 428300 - status page too slow

This means:

• When you submit a crash report you won’t have to wait longer than 30-60 seconds to view your report
• The processor now has better handling of minidump_stackwalk fatal errors
• There is an improved server status page where you can view stats on the current queue

Thanks to Lars and Aravind for getting this out the door.  The next couple of weeks will be spent improving reporter performance and closing out milestone 0.5 bugs.

History can tell you that companies don’t disclose crashes in their software. They keep a pretty close eye on what crashes and bugs are disclosed.

Mozilla doesn’t.

Rather than being the exception, openness is the rule, and that is one of the coolest things about being a part of this. My job, my everday tasks, they aren’t secret, and they are not to drive profits. They are to drive the web.

In that spirit, our crash reporting system (Socorro) is available to whoever wants to view it. Aside from user-bound statistics, crash information is available in full and anybody in the community can learn about where in the code their client crashed. They can also help provide hints or comments about what they were doing at the time they crashed.

This opens the door for the community to learn valuable things about their software and how they use it:

• What crashes the most? What crashes the most over time? What is the breakdown across branches, versions and products?
• Where did we crash? Crash signatures provide a head start for locating the cause for a crash. From there, full stack traces are available to analyze callback and find the source of the actual crash.
• What was installed? What modules were installed for a given crash? Soon we will also be able to understand what extensions were installed so we can understand the correlation between core client crashes and crashes caused by faulty extensions. The end result is a closer relationship with the extension developer community and better quality in our add-ons space.
• How are we doing? Overall the jackpot question is — are we crashing more or less? How are we doing with this beta, alpha or rc1? Are we regressing in real-life situations despite positive automated testing results??

All of this was possible because of a collaborative effort between quite a few parties:

• Mark Mentovai and the breakpad team, for writing a great client and processor under a flexible open source license that is easy to integrate
• Ted Mielczarek for his work on the client, processor and integrating the project into Firefox 3
• Benjamin Smedberg and Robert Sayre for their work in getting the initial versions of the breakpad server off the ground

Where do we go from here?

• Learn more about the project
• Send in your ideas

Of the many projects we have in 2008, this is one of the most exciting. It’s an opportunity to open up information that hasn’t historically been available to the masses, and hack on a great tool for improving the quality of all Mozilla projects