I wanted a non-painful way to figure out what’s causing bonus file IO. I’ve noticed that gtk likes to open files, but I didn’t have the exact details. So I grabbed python gdb, and with some tips on syscalls from gdb old-timers managed to produce a report to assign blame for open()ing files to relevant Mozilla functions.

Other than the gdb-hating syscalls issue, achieving this was simple

1. Compile python-enabled gdb(Next set of distribution releases should have it..I hope)
2. Define a new gdb command in a python file. I called mine “taras” for lack of a better name.
3. Set a breakpoint, attach your command to it. :
   break open
source -p /path/to/your/script.py
command 1
taras
end
4. Have the script walk the backtrace to figure out the filename and the last Mozilla function. Log the info, issue gdb continue command.
5. Print out a report and profit:
   python report()

Here is my script. The only nasty part here is that I had to read the filename out of a register (i’m on amd64, on 32 it’d be $esi instead of $rdi) because gdb doesn’t deal well with system calls.

I’ve never throught it would be this fun to use gdb. I always thought debuggers should be scriptable, thanks to Tom Tromey (lots of gdb tutorials on Tom’s blog) and any others who finally made this a reality.

David Humphrey just released his DXR bombshell. The “basic” concept behind DXR is extraction of the rich semantic information gathered by tools like GCC, Spidermonkey and xpidl. This data is then coherently linked together into a pretty UI in order to provide cleverness during code browsing sessions.

DXR will be happy to answer seemingly trivial queries:

• List implementations of interfaces in C++ (and soon JS).
• Provide relevant search results by searching semantic data first. No, grep is no longer state of the art for searching code.
• Switch between definition and declaration.
• Walk up/down class hierarchies.
• Lookup typedefs, types, etc.

I’ve been wanting to see this sort of tool built on top information exposed by Dehydra since I got it working. Words can not express how pumped I am about DXR and the magic powers that we will be granting it.

JSD Secret Sauce for JS Optimization

In general there is a misconception that implementing stuff in JavaScript will result in slower code than doing the same in C++. That may be true if the code is implemented in the exact same manner. But in real life the expressiveness and safety of a high-level language like JavaScript permits algorithmic optimizations that would often not be realistic to do in C++ because of time/safety constraints.

So how does one figure out what’s slow in JS? timeless suggested that I checkout the JavaScript Debugging API. Using the API and a small hack in spidermonkey(JSD doesn’t expose fast dom calls) I was able to hook into chrome code to get a timed trace of JavaScript functions being run.

Once I had a trace it was relatively easy to figure out to “do not call this slow thing all the time” dance (aka optimize code). I collected that work in bug 470116. Last I checked there was relatively little room for optimization left on the JS side of Fennec, so then I went to look at what’s lurking in C++.

PS. Firebug is a popular consumer of the JSD API for those times when one isn’t willing to write JS components to figure out why something is slow.

C++ Is Harder

I’ve some success with inserting probes into C++ code. I would find interesting code by running oprofile on the desktop (while doing things that I felt were slow on N810). Oprofile would then provide me a callgraph which I would visualize with this awesome little script. Then I would stick MeasurerOfTime timing blocks into interesting “hot” code and hope that I would learn something useful.

This got me thinking. Wouldn’t it be nice if there existed a JSD for C++? It’d be cool to inspect the C++ callgraph just like one does for JS. It seems like it would help on platforms that aren’t gcc and can’t inject tracing code via -finstrument-functions. Even -finstrument-functions is of limited use due to the pain of looking up symbols in shared libraries. Stay tuned.

Measuring Progress

The worst part of doing optimizations is knowing that some time in a future an innocent programmer will slightly change some seemingly innocent code and things will no longer happen quickly. Short of policing every single patch by people who previously optimized code in question there is only one thing one can do: performance tests.

Fennecmark is a benchmark for measuring responsiveness of the Fennec features that I worked on most: panning, zooming and lag during pageload. I blogged about it before. Since then Joel Maher has gotten Fennecmark to run automatically and produce results on the graph server. I think we should be logging more numbers (Tpan, Tzoom), but it’s an excellent first step in monitoring performance regressions.

Pork got reshuffled during the move, it’s now 2 repositories. oink is dead. It now depends on current versions of flex (as opposed to flex-old) and features a cleaned up buildsystem.

New way to checkout pork:

hg clone http://hg.mozilla.org/rewriting-and-analysis/pork
hg clone http://hg.mozilla.org/rewriting-and-analysis/elsa pork/elsa



cd gcc-4.3.0/.hg/patches
hg pull -u
cd ../..
hg qpush -a


and rebuild gcc to get C support in the plugin framework for GCC 4.5 (patch for trunk is in the bug).

Everything other than process_function should work in both treehydra and dehydra. Note that the objects provided by GCC will be slightly different in structure due to the underlying difference between gcc/g++, so scripts that were written for C++ may need to be updated.

This week I also got some nice contributions from a contributor with a masterplan.

Overall I expect cool things to happen once GCC 4.5 ships with plugin support. It’ll open up a jar of whoopass like C++ has not seen before.

Dehydra Docs

Benjamin redid the Dehydra documentation, it should be easier than ever to get started.

Static Analysis in Mozilla

Slowly but surely we are gathering static analysis momentum. More and more developers are pausing and thinking “I need to make sure this code has an analysis to go with it because there is no way I can prove it correct, time to write a Treehydra script…or ask someone else to make one for me”. This mindset is important for being able to introduce complex changes in a controlled manner.

On the refactoring side, things are held up a little by API compatibility. Perhaps multi-process stuff will shake things up. However, nothing can hold Chris up from letting his piglet/porky gremlins chew through locking code.

GCC 4.3.3 Is supported
GCC 4.3.3(4.3.[210] worked) broke C++ compatibility in the headers used by Dehydra.
Zach pointed out that passing -fpermissive to g++ solves the problem. Sorry to all the people who had issues building the hydras with GCC 4.3.3, that’s fixed now.

As I mentioned before, we are skipping GCC 4.4 support in Dehydra and aiming for supporting unpatched GCC 4.5. I wish that the small GCC patches were as quick to land as that big one I landed a couple of weeks ago :(.

Occasionally, it is very frustrating to see people treat open source software as immutable and independent black boxes. In my personal experience, the browser and the compiler are viewed as finished products and therefore it is OK to bitch and complain about them. That’s frustrating because the same users could be channeling that energy in a more positive way by reporting bugs, contributing code/documentation, etc.

Sometimes these rants result in rather comical conclusions: Ingo’s rant is priceless. My perspective on this:

• what have Linux kernel devs done to help GCC help them?
• <flame>Sparse is a deadend. Writing compiler code in C is silly, writing analysis code in C is sillier (and frustrating and limiting). Taking a crappy parser and bolting a crappy compiler backend onto it will result in bigger pile of crap Given how smart kernel devs are, they sure like wasting their time on crappy solutions in crappy languages.</flame>
• Wouldn’t it be cool if instead of complaining these talented people wrote a GCC plugin to do what they want?

GCC Plugin Progress

I finally landed the massively boring and annoying GTY patch. I can barely believe that the patch went in so smoothly without excess complaining from GCC devs. From GCC perspective it’s merely a cosmetic cleanup that affects a large number of headers. For us it enables Treehydra to be generated via Dehydra with little manual effort. It basically makes Treehydra possible without patching GCC. I have another 3-4 patches that need to land before trunk GCC can run the hydras out of the box. Those are mainly localized bugfixes and cleanups so I fully expect them to go in and for GCC 4.5 to rock my world.
Once GCC 4.5 ships. analyzing code will depend on a trivial matter of apt-getting(or equivalent) the hydras and specifying the analysis flags on the GCC commandline!

I tried to capture some of the “perceived performance” in numbers. My goal is to focus on user-visible areas of performance. Ideally it will enable us to track performance better to ensure that key features do not regress in performance and enable us to compare Fennec speed on various platforms. I need to spend some quality time with QA people to figure out how to achieve that.

Currently Fennecmark loads a slow-to-load webpage, zooms around it and then pans from the top to the bottom. This measures: responsiveness during pageload, zoom speed and panning lag.

See code at http://hg.mozilla.org/users/tglek_mozilla.com/fennecmark.

JSD Instrumentation

Spidermonkey provides an API that allows one to get a notification on every method entry/exit. I was able to do most of my Fennec performance analysis via a component in bug 470116. My stopwatch component times the execution of every js function call and spits out a log that has been very useful in figuring out what is taking up time in Fennec chrome.

Porkstain

I am itching to write a tool that can instrument large portions of Mozilla code such that it can be profiled across C++/JS boundaries and without any external tool support. I am guessing this would be most useful on platforms with crappy sampling tools, but it would be cool if it made finding slow codepaths easier in general. If you know any lightweight instrumentation techniques, please share.

I wrote a little prototype to insert stopwatch stuff into code deemed interesting by oprofile (stuff in the bug above). The code patching part works well, but it’s a big runtime hit and outputs too much data.

Codecon
On Sunday, I will be presenting on Pork at codecon. I have been meaning to attend codecon since the days when P2P was considered cool, was not able to make it until this year. It is a historical milestone for me. Codecon was how people at Mozilla first heard of Elsa, which is now the foundation of all our refactoring tools (it is also inherited baggage I get to maintain).

Piglet

I adopted Piglet, Dave Mandelin’s de-oinkification project, and imported it into hg. It feels really good to finally be able to do a make -j without disturbing people nearby with surprise explosions of foul language. I plan to move all relevant static analysis tools into piglet. After that I shall finally merge a dozen or so elsa repositories and end up with Pork consisting of elsa/ + piglet/.

Pork*

Chris Jones is quietly working on making Pork magnitudes more useful to average developers. It’s exciting stuff and I’ll let him announce it when he’s ready. Between his work and David Humphrey’s DXR. I think we are finally going to make it easier to hack on Mozilla for a much wider audience than before.