Before we deployed the new hardware, we had to take several steps to tame our voracious use of disk space. In the previous week, we removed the archived dumps from the database. They were rarely ever accessed but took up the lion’s share of our disk space. By migrating them to file system storage, we made a three hundred gig database migration onto new hardware into a migration of only sixty gig.

While there may be a need for tuning over the next week, Socorro users should have a much accelerated experience using the Socorro Web site.

Many thanks to aravind for shepherding this project through IT, chizu in IT for his db cloning/replication scripting/tweaking and jberkus from PostgreSQL Experts for his superior navigation skills and a steady hand at the PostgreSQL tiller.

That’s the situation that we’ve got in Socorro right now. Each time we catch a crash coming in from the field, we process it and save a “cooked” version of the dump in the database. We also save some details about the crash in other tables so that we can generate some aggregate statistics.

It’s that cooked dump that’s causing some concern. The only time that we ever access that data is when someone requests that specific crash using the Socorro UI. Considering that these cooked crashes take up nearly three quarters of the storage needs of our database, there’s not a lot of value there for the effort. They inflate the hardware requirements for our database, make backups take too long and complicate any future database replication plans that we might consider.

We’re about to migrate our instance of Socorro to new shiny 64bit hardware. Moving these great drifts of cooked dumps would take hours and necessitate potentially more than a day of down time for production. We don’t want that.

It’s time for a great migration. All those dumps are going to leave the database. We’re spooling them out into a file system storage scheme. At the same time, we’re reformatting them into JSON. In the next version of Socorro, when a user requests their dump by UUID, it will be served by Apache directly from a file system as a compressed JSON file. The client will decompress it and through javascript magic give the same display that we’ve got now.

There’s some future benefits to moving this data into a file system format. Think about all of this data sitting there in a Hadoop friendly format waiting for a future data mining project. We’ve nothing specific planned, but we’ve got the first step done.

We’re hoping to get the data migration done within the week. New versions of the processing programs will have to be deployed as well as the changes to the Web application. Once that’s done, we can proceed to the deployment of our fancy new hardware.

Our UX Engineer Neil Lee has applied some simplifications to the query form. This redesign was focused on the homepage and global navigation.

Another new feature is that MTBF and Top Crashers By Signature can be exported in CSV format. In the future, as we want to slice and dice different reports, it should be trivial to add this feature to other reports.

In addition I’ve fixed a handful of issues:

• 428110 – Quick and dirty changes to speed up crash analysis
• 478043 – Make ‘is exactly’ the default choice
• 479256 – Clarify labels to be Date Processed
• 470524 – Crash signatures not indented
• 479460 – Bad Unicode in User Comments
• 479447 – report/list with no results has JS error

We would love your feedback. Check out some recently filed bugs or send us some feedback and file a new Socorro bug.

We’ve had some known issues around MTBF and Top Crashes by Signature in the last month and are working on fixing these issues. The upside is that SeaMonkey is now in MTBF.

Our first attempt suffered from a network timeout, which required an eleven hour restore and re-run.  The re-run, done Friday, was done using a socket connection but would have required an additional 1-3 days of downtime, which was well outside our originally announced window.  Consequently, the database was rolled back to its contents as of 6:55PM PDT, January 29.  Reports have since resumed processing.

We plan on doing the following:

• Set up a complete replica of production to test this process end-to-end.  Our dry runs were done on a staging database that was roughly 1/5 the size. We anticipated a scaling of O(n), but in practice on the production server, we got performance more inline with O(n^2). So we did not expect the full extent of timeouts or how much downtime would be needed.  This will be avoided in future updates and we are setting up a stage database from a recent dump (once we gather the hardware for it).
• Push a now+ partitioning script.  The work done in bug 432450, on top of a complex migration script for old data, has logic for handling new partitions automatically which benefits new reports.  Since we don’t want to keep adding to our old database schema, we will push these updates so that new reports are properly partitioned.  Pros – in a week or two, things will be speedy and we aren’t going to struggle with timeouts.  Cons – we aren’t migrating the last 4 weeks. We will not see a performance increase when querying data older than the date of the repartitioning.

We would like to push the partitioning script (without migration of old data) on Thursday.  We will announce when it will be as soon as we know.

Long term, we are already in the process of seeking additional resources to help examine our database configuration and systems architecture.  We will have more updates on that process in the future.

Our team wants this work deployed as much as everyone else.  Thanks to everyone for their patience as we work through these issues.

• tables divided into sub tables
• queries optimized to use smaller subtables
• ad hoc queries sped up
• summary queries ignore irrelevant data
• smaller indexes
• simplified data retirement
• fresh lemon scent*

Partitioning, a feature offered by many RDMSs, is a trick to help manage titanic tables. You take a table and break it up into smaller tables, each containing a part of the whole. Say we have a data set called ‘reports’. Rather than storing them all in one table, we store them in several smaller tables. The data in each smaller table share a common trait. For example, for reports from Week 1, Week 2, and Week 3 would each have their own table. The master table ‘reports’ physically has no data in it at all. The database knows that when we reference ‘reports’ it means the union of all the smaller tables. If we do a query on the ‘reports’ table and we ask for reports from January, the database is clever enough to just look to the weekly sub-tables for January instead of looking at all the sub-tables.

We’re currently converting the Socorro database into a partitioned database (okay, technically it was already partitioned, but its partitioning was degenerate and didn’t work properly). We’re testing a Python script that is going to take the ‘reports’ table, along with the associated ‘frames’ and ‘dumps’ tables, and start breaking them into little one week chunks. Unfortunately, because of the massive size of the tables, we cannot afford to have two copies of the data in the database at the same time. The chunking of the data will be destructive. After a week of data is copied into a new partition, that corresponding week of data will be deleted from the original table.

The database, the Socorro Web App, breakpad crash processing and aggregate analysis will be down during the migration process. However, data collection will not be down: we’re not going to lose new crash data.

If we were to chunk the entire dataset, the migration process is estimated to take more than twenty hours. As a compromise, we’re going to chunk only the most recent four weeks of data and leave the rest as a single oversize partition. This will significantly reduce the time that migration takes and, therefore reduce the down time. We can get away with this because most of the aggregate reports only look at the most recent few weeks of data anyway.

Another advantage to partitioning is in the retirement of old data. In the future, we’re probably only going to keep at most one hundred twenty days of history. Any more than that and our storage needs would require its own building. To get rid of the old data, all we need to do is delete the oldest partitions. That action is fast because it doesn’t even require looking at indexes or scanning tables.

Partitioning is going to allow Socorro to scale much more smoothly. At the same time, it will make our aggregate reporting much more efficient.

This repartitioning process will happen within the next week. We will announce the scheduled down time in advance. And be assured, because of the file system changes what we made during our last big Socorro update, data collection will not be down while we’re repartitioning.

* also available in fresh wintergreen and sparkling pumpkin

Working with ss and chofman, we’ve created 2 new types of reports: a Top Crashers by Url and a Mean Time Before Failure (MTBF).

Given the current state of performance of the non-report parts of Socorro’s webapp, most of the thought and time have gone into the backend piece of these reports. You can read about the ReportDatabaseDesign on the project’s wiki.

Top Crashers by URL

On which websites do our browser builds crash the most? Which curses do our users hurl at us when this happens?

This report uses the optional url feild of a crash report to answer this question. It has two modes byurl and bydomain. You can read more about the details on TopCrashersByUrl. Crashes which have a comment, include the comment and a link to the actual crash report. Don’t worry, personal details have been removed, we don’t tie a specific user to a specific url.

We will be putting links into Socorro to these new reports, with the work neilio is doing, but for now here are various links.

We’ve enabled top crashers by URL for Firefox 3.0.5, 3.1b2, 3.1b3pre, and 3.0.6pre. Each of these link to “by domain” breakdowns, so 3.0.6pre has a link to this by domains view.

MTBF

Is this new release more crashy than previous releases?

Squeaking in before New Year’s Eve’s MFBT comes the MTBF report. It is a graph of the average number of seconds a release runs before crashing. Details are at MeanTimeBeforeFailure on the wiki.

We’re running MTBF reports for 14 releases:

Firefox major, milestone, and development releases.

Thunderbird milestone, and development releases. (No Milestone releases in Socorro yet)

Coming Soon: SeaMonkey

These reports are for a release in general as well as stats for Mac and Win, allowing for drilling down into OS. Several frontend enhancements to this report are coming.

, Product and versions in these reports include:

• Firefox 3.0.4
• Firefox 3.0.5  
• Firefox 3.1a2
• Firefox 3.1b1
• Firefox 3.1b2 
• Firefox 3.0.4pre
• Firefox 3.0.5pre
• Firefox 3.0.6pre
• Firefox 3.1b3pre
• Firefox 3.1b2pre
• Thunderbird 3.0a3

• Thunderbird 3.0b1
• Thunderbird 3.0b1pre
• Thunderbird 3.0b2pre

I’ve gotten a good dose of feedback on tweaks to make and bugs to fix, but hopefully you’ll find these new reports useful. Tomcat has already mentioned augmenting his list of urls to populate his test automation for 3.1 (using spider to test most popular urls) with the urls in these reports.

My personal goals with this redesign are to make working with crash data more efficient, and to make each screen as useful and as intuitive as possible. This project is an interesting challenge, however, as there are not a lot of publicly-available examples of crash reporting systems to use as a baseline.

“Home” page

Current page: http://crash-stats.mozilla.com/

Currently when you go to the Socorro home page you’re dumped right into search. This makes a lot of sense, but there could be a lot more useful data right up front. This wireframe tries to incorporate more of a “dashboard” approach with top crashers for release versions and other pertinent information.

Home page, default configuration

Home page with advanced filters visible

Some key changes in this wireframe:

1. Search is now called “Filter” as that’s really what you’re doing.
2. The filter options have been cleaned up a bit, and more specific filters are now grouped under Advanced and hidden by default.
3. The Advanced Filters toggle will remember if it was opened or closed, so if you regularly access any of these options they will be easily accessible.
4. There is a new “top crashers” widget that shows the top 3-5 reported crashes for current release versions.
5. The boxes underneath the filter / top crashers widget are for other chunks of data such as top crashers for development versions, mean time before failure, top URLs that cause crashes, etc.
6. The versions filter auto-fills with just the versions available for the selected product, to help keep the number of options down.
7. The “Mozilla Developers” button at the top right-hand corner opens a jump menu that lists all of the Mozilla developer web sites / tools. I think it’s kind of silly the various Mozilla developer sites aren’t linked together and this navigational tool addresses that deficiency.

Top Crashers

Current page: http://crash-stats.mozilla.com/topcrasher

At the moment the existing page is nothing more than a jumping point to link you to the various product versions. The redesigned wireframe tries to float up crash report information for more commonly used product versions while whittling the number of versions down to a more sensible number.

Individual Crash Signatures

Example: Crash reports in mozcrt19.dll@0×1838a

The redesign does away with the tabbed interface currently in use and brings everything onto one page for quicker access. The top left-hand box displays either the number of crashes by operating system (for release versions) or the number of crashes by build (for development versions).

Give us your comments, your feedback, your huddled criticism

As I mentioned these are quite preliminary and I’m very interested to hear your comments and whether you feel these wireframes are headed in the right direction.

Redesigning a tool such as Socorro is rather challenging as there aren’t many existing examples of good design in this area (or many publicly available examples at all for that matter). I also had to keep in mind that whatever is created needs to work for both Mozilla’s specific requirements as well as be generic enough to be adaptable by others, which makes this redesign even more tricky.

I spent quite a bit of time looking at similar or parallel systems such as network and web hosting dashboards to get some ideas for how this information could be displayed and these wireframes incorporate some of my discoveries.

Beefs, bouquets, comments, suggestions?

The new file system has performed quite well. The most outward expression of this improvement is the speed at which priority jobs are processed.

A priority job is any submitted crash for which someone has requested a report. There can be a backlog of submitted crashes and it might take from several minutes to several hours for the processing programs to get around to a particular job. If someone requests a particular crash, we’ve got a way for that job to jump the queue for immediate processing. Prior to the new file system, the biggest hurdle to processing a job quickly was simply finding it. There was no index to assist in find a job quickly.

The new file system changed that. All entries are indexed as they’re inserted. To see how it’s done, see my previous blog posting. This gives us very fast access to any crash dump which translates to response times of thirty to ninety seconds for priority job requests. Try it. Considering the volume of crashes we get, it’s amazing that we can zero in and process a crash so quickly.

The last two weeks hasn’t all been champagne and fireworks. We had a scare about forty eight hours after deployment. The automatic indexing scheme uses a radix algorithm to spread crash dumps evenly through a branching file system structure. During design, we chose to make this structure four levels deep. Each level did a 256 way bifurcation of the directory tree. That translates into 256^4 possible directories or about 4.3 billion. Once a directory was created, we never retired it, thinking that it would be faster to reuse old directories than to bother destroying and creating them all the time. At the rate that we received new files, we calculated that it would take years to clog up the file system. We banked on the assumption that we had at least 4.3 billion inodes available in the file system.

It was a bad assumption. It turns out that we’re using some sort of black box storage systems with variable sized inodes. We didn’t have 4.3G inodes available, we had only 64M. Back into reactive coding as performance art, we took twenty four hours to brainstorm, code, and deploy a solution. Changing the number of levels from four to three was an obvious way to reduce our foot print: 256^3 is only 16M. The number of levels of our radix directory structure is now a configuration option. The trick was making four days of data stored with four levels compatible with new data being collected with fewer levels. I managed that by encoding the number of levels into uuid of each crash.

Next time you see a crash uuid, take a look at the digits. The seventh digit from the right end will tell you how deep your crash is stored in the file system. If it’s ‘0′ then you’re stored four levels deep. Any other digit is to be taken literally: ‘2′ – two levels, ‘3′ – three levels. This crazy scheme lets the depth be switchable at run time. If directories are getting too crowded, we can raise the depth. Of it we start getting running out of inodes, we can lower the depth.

Great thanks to Frank Griswold for the coding and to Aravind for not throwing knives at me.

• Closed bugs
• Active bugs
• Code check-in RSS
• Socorro documentation

Upcoming changes:

• Socorro site redesign – Neil and Austin are working with Sam and Chris to work through crash analysis to understand how the system is used to track down the cause of certain crashes. Structure, navigation and layout will be redone to best accommodate common uses.
• Socorro database repartitioning and migration – Lars is working hard on refactoring our partitioning to better support the kinds of queries we make. The end result is better long-term maintainability and improved performance.
• New reports – MTBF reports, URL reports – Austin is working on two new report types to provide better information for crash analysis.

Current issues:

• Query timeouts – this usually happens when a user does a complex query (either no product selected or a search over a period longer than 4 weeks).
• There is no dialog saying the app is working and connections time-out leading a user to simply hit refresh to try to get where they want to go. However, many of the queries taxing the system were non-specific queries over 3-month periods. In some cases these jobs would timeout in the reporter but continue for 30 minutes or more in the database. To prevent this from happening we pushed a revision to the search form to limit query breadth.
• Search engines were crawling Socorro so we added a robots.txt file to prevent this from happening

Learn more:

• Join us in #breakpad @ irc.mozilla.org
• Checkout the code and read the documentation to setup, install and hack on Socorro.

Last week we updated the plotting library we use for graphs to use Flot and we fixed a handful of Socorro webapp bugs. Flot is a good choice where we have simple graphs to display. Below are two graphs from the crash report listing page: An new crashes by OS bar chart for production builds, and an updated crash by build, for development builds.

Also using Flot, we’ve revamped the server status page, so that we can see the health of the Socorro system. Socorro is one half to it’s sibling Breakpad which, together, provides an Open Source client and server crash reporting system. Let’s see if crash-stats is happy…

This graph is powered by a new cron job which checks the server status based on several statistics related to processing crash reports. Another cron job we’ve worked on is topcrashers.

In coming weeks we’ll be bring new life to the crash reporter web app and improving this unique and important part of how the Mozilla dev and qa teams work. We will be focusing on making it easier to use by providing new reports, easier navigation, and better visibility into crashes. What do you want to see in your crash reporting tool?