Taras’ Blog

Lately I have been focusing on optimizing Fennec startup on a delightfully inadequate platform: Windows Mobile. More on fascinating startup, performance problems and solutions later. As a result I have been doing relatively little static analysis stuff.

The main reason for taking a break is that I feel that I went from having no way to do any analysis to having production-quality tools for analysis and rewriting.  I finally have a chance to move on from developing tools to using them in everyday development. The main puzzle piece that needs completion is GCC 4.5 support in Dehydra. We are feature-complete on 4.5, just need to stabilize once the trunk stabilizes.

Drowning In Pork

A number of other people did some cool stuff in the meantime. First and foremost: Joshua Cranmer has ventured into the land of Pork and is publishing a guide to doing refactoring tools on this blog (part 1, part 2, part 3). This is cool, because until now, there were no Pork docs and nothing I write could ever match Joshua’s documenting talents.  Thanks a bunch, Joshua.

I have also received my first-ever bugfix patches to Elsa. Previously, I’ve received miscellaneous build fixes, etc, but these are the first patches that involved somebody pounding their head against the wall until they figured out why things were crashing or not accepting valid C++ code.

Introducing Dan Witte

Dan is the new static analysis go-to person. So far he facilitated an explosion of static analysis ideas (they are tracked in bug 430328). A lot of these can be expressed as <10line Dehydra analyses, so they are excellent introductory projects. If you are dying to start analyzing code, but don’t know where to begin, look in that bug. Dan has written an interesting analysis to do with finding accidental temporaries due to C++’s “wonderful” implicit conversions/etc (expect to see a blog post on that). He is also working on the holy grail of Mozilla static analysis: a full callgraph. It’s a little embarrassing that we don’t have that yet, but it’s hard and once we do have it, a whole new world of analyses will be possible.

Speaking of Callgraphs…

So while various Mozillians were pondering how awesome it would be to do inter-function analysis, an intern has beat us to writing the first useful inter-function analysis! Sully had a problem, after a tiny bit of  Dehydra coaching, he solved his problem in the amount of time it took me to eat my lunch. Brilliant! See his blog post for details. My conclusion: either Dehydra is pretty easy to use and/or we get mad genius interns .

Turns out that if someone compromises a merchant that you shop with, such as bikenashbar.com, then they are entitled to your money as long as they can show to the bank that they have your billing/shipping information. According to Chase it’s not fraud if they steal your address along with the card number.  I recommend that people use a bank that protects their customer’s finances, one that isn’t called Chase.

When it comes to taking on crazy tasks involving disguising code, you can always count on Joshua. This time Joshua has gone deep in Pork territory to do a very impressive rewrite (one that would not be even close to possible with anything other than Elsa-based tools) and he is blogging about it. He posted the first installment of his rewriting-with-Pork guide.

Still on the subject on mad csc-entist Joshua: Andrew Sutherland has finally gotten sick of crappy JS documentation tools and took matters into his own hands thanks to Joshua’s JSHydra tool. Checkout his blog post on how the documentation world will be a better place thanks to being able to build tools on top of *the JavaScript parser*.

With much pain I managed to convince a Vmware WinXP virtual machine to run in VirtualBox. At first it ran very sluggishly(>3 hours to do a build?), but after I turned off IO Apic stuff in Windows, it’s become disturbingly fast. It now takes 40minutes to build a wince fennec vs almost 80minutes it took on VMware server.

VirtualBox’s disk throughput is phenomenal, in fact, this is the first time I’ve seen almost-native speed disk io in virtual machine. A benchmark I tried reported 45mb/s 4K read/writes (these were cached on the linux side).

Unlike Vmware server, VirtualBox’s usb works without troubles, shared folders were easy to setup etc. It’s an awesome app, hope Sun/Oracle keeps it up.

Python GDB Rocks!

I wanted a non-painful way to figure out what’s causing bonus file IO. I’ve noticed that gtk likes to open files, but I didn’t have the exact details. So I grabbed python gdb, and with some tips on syscalls from gdb old-timers managed to produce a report to assign blame for open()ing files to relevant Mozilla functions.

Other than the gdb-hating syscalls issue, achieving this was simple

1. Compile python-enabled gdb(Next set of distribution releases should have it..I hope)
2. Define a new gdb command in a python file. I called mine “taras” for lack of a better name.
3. Set a breakpoint, attach your command to it. :
   break open
source -p /path/to/your/script.py
command 1
taras
end
4. Have the script walk the backtrace to figure out the filename and the last Mozilla function. Log the info, issue gdb continue command.
5. Print out a report and profit:
   python report()

Here is my script. The only nasty part here is that I had to read the filename out of a register (i’m on amd64, on 32 it’d be $esi instead of $rdi) because gdb doesn’t deal well with system calls.

I’ve never throught it would be this fun to use gdb. I always thought debuggers should be scriptable, thanks to Tom Tromey (lots of gdb tutorials on Tom’s blog) and any others who finally made this a reality.

Are you are a developer who has been frustrated by the pathetic state of the art in code search engines and code browsing experience on the web? Have you been longing for being able to view code with more aid than coloured words?

David Humphrey just released his DXR bombshell. The “basic” concept behind DXR is extraction of the rich semantic information gathered by tools like GCC, Spidermonkey and xpidl. This data is then coherently linked together into a pretty UI in order to provide cleverness during code browsing sessions.

DXR will be happy to answer seemingly trivial queries:

• List implementations of interfaces in C++ (and soon JS).
• Provide relevant search results by searching semantic data first. No, grep is no longer state of the art for searching code.
• Switch between definition and declaration.
• Walk up/down class hierarchies.
• Lookup typedefs, types, etc.

I’ve been wanting to see this sort of tool built on top information exposed by Dehydra since I got it working. Words can not express how pumped I am about DXR and the magic powers that we will be granting it.

Working on Fennec performance N810 has been very educational. I have been learning more and more about performance profiling on crappy platforms. I define a platform as crap if it has poor development tools, limited OS or other significant limitations. Linux is a crappy platform in this case because it’s running on ARM where oprofile barely does anything and there are no other performance tools for N810 (more modern ARM cpus should user-space perf counters and be more useful for instrumentation).

JSD Secret Sauce for JS Optimization

In general there is a misconception that implementing stuff in JavaScript will result in slower code than doing the same in C++. That may be true if the code is implemented in the exact same manner. But in real life the expressiveness and safety of a high-level language like JavaScript permits algorithmic optimizations that would often not be realistic to do in C++ because of time/safety constraints.

So how does one figure out what’s slow in JS? timeless suggested that I checkout the JavaScript Debugging API. Using the API and a small hack in spidermonkey(JSD doesn’t expose fast dom calls) I was able to hook into chrome code to get a timed trace of JavaScript functions being run.

Once I had a trace it was relatively easy to figure out to “do not call this slow thing all the time” dance (aka optimize code). I collected that work in bug 470116. Last I checked there was relatively little room for optimization left on the JS side of Fennec, so then I went to look at what’s lurking in C++.

PS. Firebug is a popular consumer of the JSD API for those times when one isn’t willing to write JS components to figure out why something is slow.

C++ Is Harder

I’ve some success with inserting probes into C++ code. I would find interesting code by running oprofile on the desktop (while doing things that I felt were slow on N810). Oprofile would then provide me a callgraph which I would visualize with this awesome little script. Then I would stick MeasurerOfTime timing blocks into interesting “hot” code and hope that I would learn something useful.

This got me thinking. Wouldn’t it be nice if there existed a JSD for C++? It’d be cool to inspect the C++ callgraph just like one does for JS. It seems like it would help on platforms that aren’t gcc and can’t inject tracing code via -finstrument-functions. Even -finstrument-functions is of limited use due to the pain of looking up symbols in shared libraries. Stay tuned.

Measuring Progress

The worst part of doing optimizations is knowing that some time in a future an innocent programmer will slightly change some seemingly innocent code and things will no longer happen quickly. Short of policing every single patch by people who previously optimized code in question there is only one thing one can do: performance tests.

Fennecmark is a benchmark for measuring responsiveness of the Fennec features that I worked on most: panning, zooming and lag during pageload. I blogged about it before. Since then Joel Maher has gotten Fennecmark to run automatically and produce results on the graph server. I think we should be logging more numbers (Tpan, Tzoom), but it’s an excellent first step in monitoring performance regressions.

I moved dehydra to a more official location, please update your scripts and hg settings.
New dehydra url:
http://hg.mozilla.org/rewriting-and-analysis/dehydra/

Pork got reshuffled during the move, it’s now 2 repositories. oink is dead. It now depends on current versions of flex (as opposed to flex-old) and features a cleaned up buildsystem.

New way to checkout pork:

hg clone http://hg.mozilla.org/rewriting-and-analysis/pork
hg clone http://hg.mozilla.org/rewriting-and-analysis/elsa pork/elsa


Every once in a while people want to be analyze C code. If you are one of those people, checkout bug 494960. For GCC 4.3, you’ll need to update your patch queue. If you you followed the Dehydra install instructions do something like:


cd gcc-4.3.0/.hg/patches
hg pull -u
cd ../..
hg qpush -a


and rebuild gcc to get C support in the plugin framework for GCC 4.5 (patch for trunk is in the bug).

Everything other than process_function should work in both treehydra and dehydra. Note that the objects provided by GCC will be slightly different in structure due to the underlying difference between gcc/g++, so scripts that were written for C++ may need to be updated.

I finally landed the remaining GCC patches needed to get a useful Dehydra without any of that gcc-patching pain!

This week I also got some nice contributions from a contributor with a masterplan.

Overall I expect cool things to happen once GCC 4.5 ships with plugin support. It’ll open up a jar of whoopass like C++ has not seen before.

Dehydra Docs

Benjamin redid the Dehydra documentation, it should be easier than ever to get started.

Static Analysis in Mozilla

Slowly but surely we are gathering static analysis momentum. More and more developers are pausing and thinking “I need to make sure this code has an analysis to go with it because there is no way I can prove it correct, time to write a Treehydra script…or ask someone else to make one for me”. This mindset is important for being able to introduce complex changes in a controlled manner.

On the refactoring side, things are held up a little by API compatibility. Perhaps multi-process stuff will shake things up. However, nothing can hold Chris up from letting his piglet/porky gremlins chew through locking code.