...
export LDFLAGS="-Wl,--no-keep-memory"

This tells ld to optimize for memory usage rather than speed. On my 1.5GB VM, this made a big difference because it kept ld from hitting swap as much; the link time went from ~2:30 to ~1:30. Also see https://bugzilla.mozilla.org/show_bug.cgi?id=494068.

CAVEAT EMPTOR: if you have a machine with “lots of RAM”, this flag might actually hurt link times. Also, it’s worth pointing out that the best way to speed up libxul link times is to add more RAM to your build machine, if possible. (My bright shiny new build machine can link libxul in about 5 seconds, so I don’t use –no-keep-memory anymore.)

If you want to skip the exposition and play with the code, it’s available here. Just follow the pork install instructions to get started.

Background

Porky.py started back in April when I wanted to rewrite a bunch of code. I was (well, am still, sigh) replacing a C API with a new C++ API. Basically, code that looked like this

PRLock* lock = PR_NewLock();
PR_Lock(lock);
PR_Unlock(lock);
PR_DestroyLock(lock);

was going to be changed into this

Mutex* lock = new Mutex();
lock->Lock();
lock->Unlock();
delete lock;

I quickly estimated that these old APIs were used in O(1000) places in our code, which was way more than I wanted to edit by hand. (I’m lazy, so sue me.) So I wanted an automated tool. However, this rewrite task is just beyond the reach of regular-expression-based tools like sed; existing code could do something like PR_Lock(GetStruct().GetPRLock()), which causes sed to barf. Of course there’s the pork tool, which can eat this kind of rewrite for breakfast, but looking at existing pork tools convinced me that, for this relatively simple rewrite, it was going to be a waste of work to (i) learn pork’s AST classes; (ii) learn its AST visitor idioms; (iii) learn the non-standard utility libraries pork depends on (sm::string, FakeList, …); (iv) learn pork’s patch generation library; (v) code the tool in C++ … sigh.

So I was stuck with either doing a half-assed rewrite with sed and spending a few days fixing up its mistakes, or wasting a week or so coding a pork tool. Half-assed rewrites suck. But in the bug report I filed, I realized that I was naturally describing the rewrite in a way that an automated tool could understand. I have some background in programming languages, so in the spirit of Terence Parr

“Why program by hand in five days what you can spend five yearsdays of your life automating?”

I decided to write my own tool on top of pork.

Porky.py’s specification language

Porky.py’s domain-specific language (DSL) for rewrites was designed with this use case in mind

• a C++ developer not familiar with porky.py wants to do something like my API rewrite example above
• doesn’t want to spend several days learning pork
• and doesn’t want to learn an obscure DSL
• (it’d be nice if the tool were fast, too)

So these requirements to me implied that, first, the porky.py DSL should be “minimal” in the sense of minimal additional syntax beyond C++’s — less to learn. And second, porky.py should target expression-level rewrites (as API changes usually are) rather than statement level. Statement-level rewrites complicate things.

Below is a working porky.py solution to the rewrite problem posed above. You can decide for yourself whether it meets my criteria.

rewrite SyncPrimitiveUpgrade {
  type PRLock* => Mutex*
  call PR_NewLock() => new Mutex()
  call PR_Lock(lock) => lock->Lock()
  call PR_Unlock(lock) => lock->Unlock()
  call PR_DestroyLock(lock) => delete lock
}

The rewrite rule type PRLock* => Mutex* means: everywhere the type “PRLock*” appears, change it into “Mutex*”. The second kind of rule here, PR_Lock(lock) => lock->Lock() is more interesting; it means that, at any callsite matching

PR_Lock($lock$)

where $lock$ is any expression, change this line into

$lock$->Lock()

These kinds of rules are porky.py’s big advantage over sed et al.: because pork.py has access to a C++ AST through pork, it can match patterns that require strictly more power than regular expressions provide. One can write rules like call Foo(a, b, c) => c.Method(b, a), and the rule will rewrite call sites like Foo(x().y().z(), r(s(t)), u.v.w()) into u.v.w().Method(r(s(t)), x().y().z()).

And finally, porky.py provides the creature comfort of one-liner shell invocations for really simple rewrites

porkyc -e 'call SomeFun(a, b) => OtherFun(b, a)'

After which the compiled pork tool can be invoked. (Docs forthcoming on MDC.)

Code rewriting workflow when using porky.py

After writing porky.py, I used it to edit a large quantity of code in a couple of hours. These patches haven’t all made it into mozilla-central yet (for a variety of reasons), but I wanted to show the steps I took to generate them. This will eventually find its way into an MDC guide.

$ porkyc -m sync_primitive_upgrade.porky
  (outputs and compiles code in |SyncPrimitiveUpgrade.code/|)
$ SyncPrimitiveUpgrade.code/dorewrite ~/mozilla-code/*.ii -x *nspr* > mozilla-code.patch
   (does n-way parallel rewrite on matching files; n depends on your system)
   (doesn't include files matching the pattern *nspr* in the patch)
   (writes patch to stdout)

Next, I would apply this patch and compile, fixing up problems by hand (hey, porky.py is a prototype). Then it was hg qdiff and the patch was up for review. I could generate these patches much faster than they would have been able to be reviewed; this ended up being the bottleneck.

Eventual goal for porky.py

I’d like it to support this rewrite

rewrite FooToBar {
  class Foo => Bar {
    member mMember => member_
    method Method(a1, a2) => method(a2, a1)
  }
}

which would entail

• rename class Foo into class Bar
• rename type “Foo” into type “Bar” (including Foo*, Foo&, …)
• change calls to Foo constructors into Bar constructors
• rename declaration of Foo.mMember into Bar.member_
• convert accesses of ((Foo)inst).mMember into ((Bar)inst).member_, (and similary for inst->mMember, …)
• rename declaration of Foo::Method into Bar::method
• rename implementation of Foo::Method into Bar::method
• convert calls to ((Foo)inst).Method(a1, a2) into ((Bar)inst).method(a2, a1) (and similary for inst->Method(a1, a2), … and similary for subclasses of Foo …)

I should note that having porky.py rewrite declarations and definitions is not so important (though it would be nice!): there is only one declaration/definition. Rewriting uses is much more important, since there are any number of uses.

I won’t implement this kind of rewrite until I need it. Sorry! But please feel free to dive in to the porky.py code and do it yourself!

How porky.py fits into the “rewrite tool space”

Rewrite tools have to trade off several factors. It’s good to have a small, familiar DSL, as these are easier to learn and remember. But it’s also good to have a large and expressive DSL, for raw rewrite power. The table below is my attempt to fit porky.py into the space of relevant rewrite approaches I’m aware of. It compares porky.py with

• pork. Rewrites specs are written in C++, which is not obscure to C++ programmers. Rewrite specs are very verbose. Any possible rewrite of C++ code can be expressed in pork. Pork is very fast.
• XML+XSLT. Rewrite specs are written in XML/XSLT modeled on a particular C++ AST; very obscure. Rewrite specs are relatively concise. Any possible rewrite can be expressed. Very slow.
• Tree transformation (e.g. in ANTLR). Very obscure DSL. Rewrite specs are relatively concise. Can express (usually) any possible rewrite. Usually relatively slow.
• Coccinelle/SmPl. DSL relatively familiar. Rewrite specs concise. Can express most statement-level rewrites. Can be fast.
• porky.py. DSL familiar. Rewrite specs concise. Express some expression-level rewrites. Fast.
• sed. DSL familiar. Specs concise. Extremely limited power. Very fast.

  	            <---- More LoC(bad) ----- Fewer LoC (good) --->
                    <-- Less obscure(good) -- More obscure(bad) -->

	               No DSL  |  Some DSL  |  More DSL  |  All DSL
                     +---------+------------+------------+-----------
        All possible |  Pork   |    ???     |    ???     |  XML+XSLT/tree trans.
	  ~Statement |         |            |coccinelle* |
	 ~Expression |         |  porky.py  |            |
	       Names |         |            |            |
 Crappy rename hacks | sed**   |            |            |
	-------------
 	 * only works for C code
	 ** assumption: regular expressions are well-known enough that
	    negative "DSL" connontations don't apply.

To be honest, the biggest lesson I learned from this project is that pork can be a good lower-level “engine” for higher-level tools. I didn’t know about Coccinelle when I wrote porky.py; if I had, I might have tried retargeting it to pork instead of starting from scratch. I prefer some of porky’s syntax/semantics to Coccinelle’s, but I think the additional complexity of SmPL adds a compelling amount of power over porky.py’s simpler DSL. Upgrading SmPL to parse C++ and retargeting it to pork might be a good project for someone else.

But of course, since Coccinelle doesn’t understand C++, we’re “stuck” with porky.py for the foreseeable future .

Porky.py for language nerds

Porky.py is implemented as a relatively simple source-to-source translator written in Python. It converts porky.py specifications into a C++ header containing rewrite rules defined in a sort of “bytecode.” This header is included by a general porky.py C++ tool that uses pork. This tool “interprets” the rules, and if a rule matches part of the C++ AST, the tool generates a patch hunk according to the porky.py spec. This is fairly similar to how a regular expression engine would implement a “replace” function, although the matching is obviously fairly different.

There were a few interesting problems that arose while I designed the porky.py language. The first was what the semantics of rule matching should be. The issue is that multiple rules can match the same program text. For example, in the spec

call Foo => Bar
  (means "rewrite all calls to Foo into Bar, regardless of arguments")
call Foo(a, b) => Bar(b, a)
  (means "rewrite only the two-argument version of Foo into Bar, reverse the args")

both rules will match Foo(1, 2). Which should be used?. My solution was use the most “specific” rule. I defined what “specific” meant by the following rules. First, a call pattern with arguments is “more specific than” a call pattern without arguments. And second, a “literal” pattern is “more specific than” a wildcard pattern. For example, Foo::kSomething is more specific than f. Porky.py implements these heuristics by ordering all the rewrite rules at compile time by decreasing “specificity” and then attempting matches in that order. The first rule to match is chosen for the rewrite.

A second issue that arose was how rewrite specifications should express “literal” patterns — i.e., match this exact text — vs. “wildcard” patterns — i.e., match any expression in this syntactic slot. The problem arises in this rule call Foo(Bar, a) => Baz(a, Bar). Are “Foo”, “a”, and “Bar” literals or wildcards? I didn’t want to add special syntax for wildcards because of the “minimal DSL” design goal. So, my solution was to be as “greedy” as possible about choosing wildcard variables; I think this is likely to be least surprising. In the example above, “Foo” shouldn’t be a wildcard because if it were, it would match any function call with two arguments. That would be silly. But, both “a” and “Bar” are wildcards; in fact, any identifier used as a C++ “expression variable” (other than function names) are treated as wildcards. The “escape hatch” is C++ namespace qualification; if in the first example “Bar” was meant to only match some global symbol “Bar”, then the pattern could have been written as call Foo(::Bar, a). There are numerous other heuristics possible here, and I may change porky.py’s depending on feedback.

A third issue was how porky.py expressions should be typed, i.e., what their C++ type should be. For example, this porky.py rule seems innocent enough call foo->Bar() => foo->Baz() when one is thinking “foo is a Foo instance,” but how can porky.py glean that information? (Note that this is not a problem for the RHS “foo”, since its type is already known by the time porky.py is ready to generate a patch hunk.) My solution to this was to have porky.py programmers write these LHS method calls in desugared form; in the example above, call Foo::Bar(foo) => foo->Baz(). Eventually, though, I would like to use the class syntax I introduced above to resolve this ambiguity. For example: class Foo { call Bar => call Baz } (note, though, that this isn’t implemented yet). C++ inheritance also complicates things here. When implementing porky.py I punted on inheritance (hey, it was five day’s work!), but I think the problems it presents have reasonable solutions.

Hacking porky.py

If you’re interested in taking up any of the porky.py extensions I suggested here, or just want some porky.py technical support, send me an e-mail at cjones@mozilla.com or drop by the #static channel at irc.mozilla.org.

First, I browse around. Nothing particularly exciting there, except that two Firefox programs are running — Firefox itself, and gecko-iframe. The second program is new: it’s drawing the web pages to the screen. Currently in Firefox, this is all done within the same program.

The fun comes when I kill -9 this gecko-iframe, the “tab” containing mozilla.com. To the non-geeky, invoking kill -9 on a program causes it to crash IMMEDIATELY. This simulates what would happen if, say, you tried to run a buggy plugin and it got itself into trouble. Notice that only the “content” disappears when the page crashes; the user interface itself keeps running as if nothing happened. This is a big step forward! If I were to kill -9 the current version of Firefox, everything would die, user interface and tabs.

With Firefox protected from buggy pages and plugins, more fun is possible. This video shows me pressing a “Recover” button that relaunches the page that just crashed. There are many more possibilities for recovering from these errors, and I’m excited to see what our user interface folks cook up.

This demo shows off a lot of hard work from Ben Turner, Benjamin Smedberg, Boris Zbarsky, and Joe Drew. (We also have plugins running in their own, separate processes, in an incomplete way. However, the plugins still refuse to draw to the screen and so wouldn’t make for a very good demo.) Drop by #content on IRC or read mozilla.dev.tech.dom if you want to find out more details of what the Electrolysis team is up to.

The carrots

We’ve improved the API in the following ways

• all classes are in the mozilla namespace
• the “bare” primitives (Mutex, Monitor, CondVar) can now be allocated on the stack or as direct class members. Their constructors use a proto-non-failing malloc, so your code will never see out-of-memory errors.
• the primitives and their RAII wrappers (MutexAutoLock/Unlock, MonitorAutoEnter) provide methods to assert that the primitive is “owned,” or not, by the currently executing thread. These should be used liberally in your code to enforce locking invariants (the assertions disappear in optimized builds).
• the deadlock detector used by the new primitives in debug builds is asymptotically (and practically) more efficient than the old, should catch more deadlocks, and is now thread safe (!).

The sticks

As with any change to core APIs, this one will require a lot of rewriting. We’ll keep the old one around for a while, but new patches should use the new API. (Aside: Shawn Wilsher has been an early adopter, for the storage code. Check out what he’s been doing for some real-world examples.)

We’ve been extending our static analysis/refactoring tools to make upgrading old code easier. Hopefully, few people will need to worry about this. I’ll soon be blogging about our work; I think it’s a good example of how sweeping changes to existing code can be done quite easily, by one or two guys, in a matter of weeks.

The main change to the API that may cause some consternation is the removal of the nsAutoLock::lock/unlock and nsAutoMonitor::Enter/Exit methods from MutexAutoLock and MonitorAutoEnter. It has been decreed that they are bad news. Unfortunately, these are fairly difficult to rewrite automatically, so we may need to recruit a few module owners to help out.

Further reference

Most of the new API is documented on MDC. But please ping cjones on IRC if you have more questions or concerns.

[EDIT I see from PMO that I should be h4 and below.]