The new file system has performed quite well. The most outward expression of this improvement is the speed at which priority jobs are processed.

A priority job is any submitted crash for which someone has requested a report. There can be a backlog of submitted crashes and it might take from several minutes to several hours for the processing programs to get around to a particular job. If someone requests a particular crash, we’ve got a way for that job to jump the queue for immediate processing. Prior to the new file system, the biggest hurdle to processing a job quickly was simply finding it. There was no index to assist in find a job quickly.

The new file system changed that. All entries are indexed as they’re inserted. To see how it’s done, see my previous blog posting. This gives us very fast access to any crash dump which translates to response times of thirty to ninety seconds for priority job requests. Try it. Considering the volume of crashes we get, it’s amazing that we can zero in and process a crash so quickly.

The last two weeks hasn’t all been champagne and fireworks. We had a scare about forty eight hours after deployment. The automatic indexing scheme uses a radix algorithm to spread crash dumps evenly through a branching file system structure. During design, we chose to make this structure four levels deep. Each level did a 256 way bifurcation of the directory tree. That translates into 256^4 possible directories or about 4.3 billion. Once a directory was created, we never retired it, thinking that it would be faster to reuse old directories than to bother destroying and creating them all the time. At the rate that we received new files, we calculated that it would take years to clog up the file system. We banked on the assumption that we had at least 4.3 billion inodes available in the file system.

It was a bad assumption. It turns out that we’re using some sort of black box storage systems with variable sized inodes. We didn’t have 4.3G inodes available, we had only 64M. Back into reactive coding as performance art, we took twenty four hours to brainstorm, code, and deploy a solution. Changing the number of levels from four to three was an obvious way to reduce our foot print: 256^3 is only 16M. The number of levels of our radix directory structure is now a configuration option. The trick was making four days of data stored with four levels compatible with new data being collected with fewer levels. I managed that by encoding the number of levels into uuid of each crash.

Next time you see a crash uuid, take a look at the digits. The seventh digit from the right end will tell you how deep your crash is stored in the file system. If it’s ‘0′ then you’re stored four levels deep. Any other digit is to be taken literally: ‘2′ – two levels, ‘3′ – three levels. This crazy scheme lets the depth be switchable at run time. If directories are getting too crowded, we can raise the depth. Of it we start getting running out of inodes, we can lower the depth.

Great thanks to Frank Griswold for the coding and to Aravind for not throwing knives at me.

Indexing by a single key is simple in a file system. The path is the key to a file. Indexing by a second key in a file system requires links. We chose to use symbolic links as our pointers to indicate relationships between files. There’s an interesting thing about symbolic links: if the path they point to isn’t too long, the path is stored right with the inode. Reading the target path from a symbolic link is fast. We created two radix sort hierarchies, one for the name and one for date. We store our uuid named data files on the name branch. Then we have symbolic links span the gap between the branches to indicate the time interval in which a file was submitted.

Our applications use these two hierarchies in a certain order every time. On first reading the file system, we’re looking only for dumps that we haven’t seen before. We walk the hierarchy on the date side and we know that every symbolic link we encounter represents a new crash dump. We pass the uuid on to the next application for processing and then we remove the symbolic links. That guarantees that any link we find on the date side is always new. From then on, we only will need to lookup that uuid by name. The remaining hierarchy of the name branch is optimized for that.

We also use this hierarchy for long term storage of crash dumps using a different file system root. Using the name branch, we can look up files by uuid very quickly even if the number of stored files is huge. When it’s time to retire old data that is no longer needed, we can walk the date branch to find and delete only items older than a threshold.

We expect this new system to speed up priority processing significantly. The uniform API embodied by our new class JsonDumpStorage will increase reliability and consistency across the applications that use the file system.

For more information, see: SocorroFileSystem
For progress information on deployment: Bug 458798

• main monitor thread takes > 1 hr to complete a full scan of pending jobs on disk
• priority job processing depends on this thread

In order to fix these delays, we have made priority job monitoring a separate thread from the main queue thread, but we are working to reduce the delay back to 1-2 minutes (which is what we’re used to!).

Currently, we are blocked by issues with this new method related to filesystem scanning.  This is blocking the archiving and data re-import mentioned in our last set up updates.  See the related code.

We are working to fix this asap, and will provide updates this evening.

• Report data prior to June 23rd will be temporarily unavailable.
• New reports will work as usual on our new database.
• Old data will be imported into the new database using an improved partitioning plan.

Reasons why we need to do this now:

• Firefox 3 crash throughput has been about triple the projected amount (due to the popularity of Firefox 3).
• The current database has a partition that is unmanageable without significant downtime (2-3 days)
• The Socorro reporter is not responsive because of the size of the current partition.

No individual reports will be lost, they will be restored over the next week. Please let us know if you have any questions.

• Bug 426940 – Reduce or eliminate delay in collector to monitor hand-off
• Bug 426940 – Fix processor handling of error conditions
• Bug 428300 – status page too slow

This means:

• When you submit a crash report you won’t have to wait longer than 30-60 seconds to view your report
• The processor now has better handling of minidump_stackwalk fatal errors
• There is an improved server status page where you can view stats on the current queue

Thanks to Lars and Aravind for getting this out the door.  The next couple of weeks will be spent improving reporter performance and closing out milestone 0.5 bugs.

Mozilla doesn’t.

Rather than being the exception, openness is the rule, and that is one of the coolest things about being a part of this. My job, my everday tasks, they aren’t secret, and they are not to drive profits. They are to drive the web.

In that spirit, our crash reporting system (Socorro) is available to whoever wants to view it. Aside from user-bound statistics, crash information is available in full and anybody in the community can learn about where in the code their client crashed. They can also help provide hints or comments about what they were doing at the time they crashed.

This opens the door for the community to learn valuable things about their software and how they use it:

• What crashes the most? What crashes the most over time? What is the breakdown across branches, versions and products?
• Where did we crash? Crash signatures provide a head start for locating the cause for a crash. From there, full stack traces are available to analyze callback and find the source of the actual crash.
• What was installed? What modules were installed for a given crash? Soon we will also be able to understand what extensions were installed so we can understand the correlation between core client crashes and crashes caused by faulty extensions. The end result is a closer relationship with the extension developer community and better quality in our add-ons space.
• How are we doing? Overall the jackpot question is — are we crashing more or less? How are we doing with this beta, alpha or rc1? Are we regressing in real-life situations despite positive automated testing results??

All of this was possible because of a collaborative effort between quite a few parties:

• Mark Mentovai and the breakpad team, for writing a great client and processor under a flexible open source license that is easy to integrate
• Ted Mielczarek for his work on the client, processor and integrating the project into Firefox 3
• Benjamin Smedberg and Robert Sayre for their work in getting the initial versions of the breakpad server off the ground

Where do we go from here?

• Learn more about the project
• Send in your ideas

Of the many projects we have in 2008, this is one of the most exciting. It’s an opportunity to open up information that hasn’t historically been available to the masses, and hack on a great tool for improving the quality of all Mozilla projects

• refactor of processor code, which is 1/3 of the breakpad server
  architecture
• update of reporter to allow for instant queuing of requested reports

This means:

• If you submit a crash, going to that crash page will:
  • Show you a “haven’t queued it yet” page instead of a 404
    page that will update in < 10 min
  • Once queued, you’ll see a “report pending” page that will
    redirect to the finished report in < 21 seconds
• Wait time for reports from testers is reduced to 10 min max,
  sometimes 21 seconds best-case
• We are working on eliminating the 10 min portion but there are
  reasons why we can’t spam the monitor that is responsible for
  queuing new reports that are on disk — more on that next week (I
  want this to get down to: load, wait 20 seconds, BAM! see your report)

Thanks for everyone’s patience with the crash report backlog during releases — we hope this helps many of you.

Let me know if you have any questions.  More to come in the next few weeks!  Thanks to Lars, Ted and Aravind for their help with developing/ testing and pushing these updates.

• Graphs to show crash population over time
• Crash reports by operating system
• Crash reports by build and operating system

There are plenty of fun graphs that show trends over time. For example, see this graph that shows a common crash in gfxFont::Draw:

Up next, we’ll be working on finishing up milestone 2 items and work towards milestone 3. For more about the project and what we’re working on:

• See the project home page
• List of current to-dos (milestone 2)
• All open issues
• Our wiki page discussing requirements and design

If you have any comments please find us in mozilla.dev.quality or #breakpad@irc.mozilla.org on IRC!

• You really have to read the “caveats” part of the manual (scroll down, very bottom)
• Server config matters (SET constraint_exclusion = on;) if you want to avoid unnecessarily checking out-of-bounds partitions
• Child tables don’t inherit permissions, so owning the parent table won’t automagically grant you access to child tables — which in this case prevented me from setting up my explicit indexes.
• Foreign key constraints on a parent table are not inherited. This is actually a bug in pgsql. They plan on changing “CREATE TABLE LIKE” in pgsql 8.3 but fixing this for INHERIT statements is a bit different since it means inheriting for the lifetime of the parent->child relationship (instead of one-time-only during creation). For more, read chizu’s thread on the issue — he actually submitted a patch that fixed things for table creation.

Long story short, partitioning is tricky business. For more, see bsmedberg’s blog, where he talks about query performance in queries spanning multiple partitions and his battle with the postgresql query planner.

From the very beginning, even before you start messing with the query planner, I think it’s important to understand the pitfalls of inheritance in pgsql and how it will affect your use of partitions.

So just be extra careful when you read the manual and make sure you’re aware of these issues when you plan your schema. If you’re not, it’ll come back to bite you.