Taras’ Blog

I spent couple of days fixing the remaining test-suite failures on GCC 4.5 trunk for Dehydra. Since the last time I looked into this, GCC went from crashing all over the place to only crashing if I did something bad. It was nice to discover that as a result of switching to 4.5 Dehydra users will get saner .isExplicit behavior and more precise location info.

Treehydra will take more work due to me misunderstanding GTY annotations.

By the way, I am really grateful for all of the people who contributed GCC 4.5 fixes so far. You guys have been a big help in getting Dehydra testsuite to 100% on 4.5. Looks like I will meet my goals to finish De+Treehydra by the end of the year in time for GCC 4.5 release and my “Introducing Dehydra to the Developer World”-type talk at LinuxConf.au.nz 2010.

Startup
I reduced my focus on startup speed at the moment to catch up on Dehydra. I plan to work on reducing xpconnect overhead during startup next, ie more of this bug.

Last week I was in Canada to present at FSOSS with David Humphrey on awesome Mozilla Tools: Dehydra, DXR, Pork, etc. I think we managed to convey the message regarding what a sad affair that current developer development tools are.

General-Purpose Dehydra Scripts

Dehydra grew out of Mozilla’s constant need to figure out what is going on in the source code. As a result most of our scripts are very Mozilla API-specific. This makes harder for people outside of Mozilla to learn Dehydra. There is no library of Dehydra code that one can just plugin to start analyzing their codebase. Instead one has to sit down, figure out what Dehydra is capable of and then see if any of the problems facing the developer can be solved this way. If anyone wants to contribute such a library, let me know.

In the meantime, more general-purpose analyses are surfacing.

Shadowed Members

My favourite script so far is the member-shadowing checker. I ran into a member-shadowing warning that is unique to Sun’s C++ compiler. It was triggered by some code that I just landed on the tree. I fixed the warning, but within a few days a coworker ran into a bug caused by that member shadowing(due to having an unlucky revision of the code). The following example shows how simple it was to implement the warning in GCC/Dehydra.

See bug 522776 for the complete story on adding the member shadowing check to Mozilla.

Printf
Another general purpose analysis was done outside of Mozilla by Philip Taylor for his game. His script checks wide printf format strings (which are overlooked by gcc).
Independently, Benjamin wrote a printf checker for Mozilla printf-like code, see bug 493996.

Custom Sections in Object Files
We have long speculated about how nice it would be if Dehydra could emit info into object files that could then be yanked out of the resulting binary (by say, valgrind). bug 523435 will soon make that a reality.

Update:
Two photos from FSOSS.

While writing C++ sometimes one wishes that one could squeeze a little more out of the type system. In this particular case, Zack Weinberg (layout-refactorer extraordinaire), wanted to make sure that certain methods always get overridden in derived classes. Unfortunately, in that particular design, those methods were not pure-virtual. At this point most C++ hackers would cry a little and move on without any compiler assistance.

Instead of crying, Zack added a NS_MUST_OVERRIDE attribute to methods along with a matching Dehydra script. See the source code and the bug for how simple it can be to extend C++ with a useful new check.

Nothing makes me happier than seeing developers land big code changes and accompany them with compiler checks instead of relying on programming folklore to maintain important invariants.

Lately I have been focusing on optimizing Fennec startup on a delightfully inadequate platform: Windows Mobile. More on fascinating startup, performance problems and solutions later. As a result I have been doing relatively little static analysis stuff.

The main reason for taking a break is that I feel that I went from having no way to do any analysis to having production-quality tools for analysis and rewriting.  I finally have a chance to move on from developing tools to using them in everyday development. The main puzzle piece that needs completion is GCC 4.5 support in Dehydra. We are feature-complete on 4.5, just need to stabilize once the trunk stabilizes.

Drowning In Pork

A number of other people did some cool stuff in the meantime. First and foremost: Joshua Cranmer has ventured into the land of Pork and is publishing a guide to doing refactoring tools on this blog (part 1, part 2, part 3). This is cool, because until now, there were no Pork docs and nothing I write could ever match Joshua’s documenting talents.  Thanks a bunch, Joshua.

I have also received my first-ever bugfix patches to Elsa. Previously, I’ve received miscellaneous build fixes, etc, but these are the first patches that involved somebody pounding their head against the wall until they figured out why things were crashing or not accepting valid C++ code.

Introducing Dan Witte

Dan is the new static analysis go-to person. So far he facilitated an explosion of static analysis ideas (they are tracked in bug 430328). A lot of these can be expressed as <10line Dehydra analyses, so they are excellent introductory projects. If you are dying to start analyzing code, but don’t know where to begin, look in that bug. Dan has written an interesting analysis to do with finding accidental temporaries due to C++’s “wonderful” implicit conversions/etc (expect to see a blog post on that). He is also working on the holy grail of Mozilla static analysis: a full callgraph. It’s a little embarrassing that we don’t have that yet, but it’s hard and once we do have it, a whole new world of analyses will be possible.

Speaking of Callgraphs…

So while various Mozillians were pondering how awesome it would be to do inter-function analysis, an intern has beat us to writing the first useful inter-function analysis! Sully had a problem, after a tiny bit of  Dehydra coaching, he solved his problem in the amount of time it took me to eat my lunch. Brilliant! See his blog post for details. My conclusion: either Dehydra is pretty easy to use and/or we get mad genius interns .

Are you are a developer who has been frustrated by the pathetic state of the art in code search engines and code browsing experience on the web? Have you been longing for being able to view code with more aid than coloured words?

David Humphrey just released his DXR bombshell. The “basic” concept behind DXR is extraction of the rich semantic information gathered by tools like GCC, Spidermonkey and xpidl. This data is then coherently linked together into a pretty UI in order to provide cleverness during code browsing sessions.

DXR will be happy to answer seemingly trivial queries:

• List implementations of interfaces in C++ (and soon JS).
• Provide relevant search results by searching semantic data first. No, grep is no longer state of the art for searching code.
• Switch between definition and declaration.
• Walk up/down class hierarchies.
• Lookup typedefs, types, etc.

I’ve been wanting to see this sort of tool built on top information exposed by Dehydra since I got it working. Words can not express how pumped I am about DXR and the magic powers that we will be granting it.

I moved dehydra to a more official location, please update your scripts and hg settings.
New dehydra url:
http://hg.mozilla.org/rewriting-and-analysis/dehydra/

Pork got reshuffled during the move, it’s now 2 repositories. oink is dead. It now depends on current versions of flex (as opposed to flex-old) and features a cleaned up buildsystem.

New way to checkout pork:

hg clone http://hg.mozilla.org/rewriting-and-analysis/pork
hg clone http://hg.mozilla.org/rewriting-and-analysis/elsa pork/elsa


How well are you packing your structs?
Arpad asked that question with an awesome Dehydra script and came up with an interesting list.

GCC 4.3.3 Is supported
GCC 4.3.3(4.3.[210] worked) broke C++ compatibility in the headers used by Dehydra.
Zach pointed out that passing -fpermissive to g++ solves the problem. Sorry to all the people who had issues building the hydras with GCC 4.3.3, that’s fixed now.

As I mentioned before, we are skipping GCC 4.4 support in Dehydra and aiming for supporting unpatched GCC 4.5. I wish that the small GCC patches were as quick to land as that big one I landed a couple of weeks ago .

I feel strange working on GCC-specific stuff and then discussing it on planet mozilla as mozilla work. However, without GCC, Dehydra and Treehydra would not be half as awesome (much less feasible even). The power of open source is that it allows us to leverage the entire open source ecosystem to achieve specific goals. When open source projects combine their efforts, not even the biggest software companies can compete as cross-project goals would be incredibly expensive and unpleasant otherwise.

Occasionally, it is very frustrating to see people treat open source software as immutable and independent black boxes. In my personal experience, the browser and the compiler are viewed as finished products and therefore it is OK to bitch and complain about them. That’s frustrating because the same users could be channeling that energy in a more positive way by reporting bugs, contributing code/documentation, etc.

Sometimes these rants result in rather comical conclusions: Ingo’s rant is priceless. My perspective on this:

• what have Linux kernel devs done to help GCC help them?
• <flame>Sparse is a deadend. Writing compiler code in C is silly, writing analysis code in C is sillier (and frustrating and limiting). Taking a crappy parser and bolting a crappy compiler backend onto it will result in bigger pile of crap Given how smart kernel devs are, they sure like wasting their time on crappy solutions in crappy languages.</flame>
• Wouldn’t it be cool if instead of complaining these talented people wrote a GCC plugin to do what they want?

GCC Plugin Progress

I finally landed the massively boring and annoying GTY patch. I can barely believe that the patch went in so smoothly without excess complaining from GCC devs. From GCC perspective it’s merely a cosmetic cleanup that affects a large number of headers. For us it enables Treehydra to be generated via Dehydra with little manual effort. It basically makes Treehydra possible without patching GCC. I have another 3-4 patches that need to land before trunk GCC can run the hydras out of the box. Those are mainly localized bugfixes and cleanups so I fully expect them to go in and for GCC 4.5 to rock my world.
Once GCC 4.5 ships. analyzing code will depend on a trivial matter of apt-getting(or equivalent) the hydras and specifying the analysis flags on the GCC commandline!

The nice folks at FSF allowed GCC have plugins. In a couple of GCC releases, Dehydra(4.5 if we are lucky) will work with distribution GCCs. Of course the API is yet to be decided on, but we have been coordinating with authors of other GCC plugin efforts to ensure that the final API meets reasonable needs.

In the future enabling static analysis checks will involve little more than specifying –with-static-checking in your Mozilla build!

JSHydra

The other breakthrough news is that Joshua Cranmer has been working on hooking up a *hydra style API to the Spidermonkey parser. This resulted in JSHydra. Ability to look into JavaScript has been sorely missing from our stack, so this is extremely exciting.

Some time ago, Igor mentioned that there is code in SpiderMonkey that pleads to the programmer that from a certain point in a function code must flow through a label(ie a finalizer block). Treehydra made it to possible to turn that weak plea into an error message when static checking is enabled. See the bug for more details. My favourite static analyses are all about turning informal “gurantees” into angry compiler complaints.

This is my first static analysis that landed in the mozilla-central tree. It’s also the simplest one and may be a decent starting point for solving similar problems. I’d be cool to see this particular feature utilized outside of SpiderMonkey. Unlike human-powered code-inspection, it excels at finding accidental early returns covered up by macros.