Rooftop lab

This is my first post about Harmony modules. I hate to start right out of the gate with a long boring post about timing details, but it seems the few people who might eventually be interested in it are actually interested right now! So let’s dive in.

Once we have modules, running a script will sometimes hit the network.

    <script>
        module blottr from "blottr.js";
    </script>

(By the way: you will have to opt in to Harmony syntax somehow—yet to be determined. This post will ignore that consideration.)

When the HTML parser reaches this script, parsing will pause until “blottr.js” is loaded. This matters if “blottr.js” touches the DOM during its initialization, for example. In terms of timing, it behaves just like a <script src=> element.

(As a performance optimization, we plan to change the HTML5 parser to skim ahead, find probable module statements, and pre-fetch the scripts. Again, just like <script src=>.)

But what about this?

    function silly() {
        eval('module blottr from "blottr.js";');
    }

What should happen if silly() is called from an event handler? Should it block until “blottr.js” comes in from the network? That would be a synchronous, blocking call, like synchronous XHR.

That would be gross. So in Harmony, eval will reject such code with a SyntaxError. In short, if a JavaScript caller is waiting, the module X from "url" syntax is banned. If your code needs a module off the network, you’ll just have to run it asynchronously somehow. Two new APIs, SystemLoader.load(urlcallback, errorCallback) and SystemLoader.asyncEval(code, callback, errorCallback), are proposed for that.

Now. Suppose we take the first example above and put it in a document.write call.

    <script>
        document.write(
            '<script>' +
            '    module blottr from "blottr.js";' +
            '</scr' + 'ipt>');
    </script>

Now what happens? (Benjamin Smedberg brought this case to my attention about 12 seconds after I described modules to him. It might have been less.)

To me, this particular flavor of document.write() weirdness was something new. I’m still not totally sure how we will handle it. Boris Zbarsky suggested again treating it the same as a <script src=> element: silently make that script load asynchronously, just because document.write() created it. That sounds plausible enough.

There are still a few more special cases to sort through. What happens if you try to load a module by assigning a string to an event handler attribute of a DOM element? I’m not sure yet. Perhaps we will throw a SyntaxError.

Bored to tears? Sorry! I’ll write again in a few days, explaining what Harmony modules are, who’s working on them, and why.

Back in April, Jim Blandy and I started hacking on a little project in a user repo.

Last Wednesday, without warning, I posted the results of three months of work for review in bug 672829. The whole patch is 336KB (actually 534KB, if you count tests), so I split it up into sections and requested code reviews from eight different JS hackers.

The first review came in before I was even done posting all the patches.

That afternoon, I sent a brief e-mail to the JS team asking for help getting the reviews done quickly. That was the only thing I did that I wouldn’t normally do. By noon Friday, just 48 hours later, twelve of the thirteen reviews had been granted, and Brendan was about halfway through the thirteenth.

The reviewers were:

David Anderson
Andrew Drake
Brendan Eich
Andreas Gal
Blake Kaplan
Bill McCloskey
Luke Wagner
Jeff Walden

Thanks, guys.

The best part: this is no fluke or one-off effort. It’s like this every day in js/src. How did this happen? Why do you get faster code reviews in the JavaScript engine than anywhere else in the project?

I really don’t know, but I have some guesses. Brendan Eich was the module owner for many years, and he always turned reviews around lightning-fast—and not by skimming, either, as you know if you’ve ever read a /be review. (You can read a totally typical one in this bug.) I think Rob Sayre probably had an influence as well. Maybe when you run a team with that “hey, are we all acting like adults here, and if not, why not” attitude for a few years, you get a culture of fast reviews.

Whatever the reason, I’m grateful. Fast reviews make me more effective. Some days, they make my job really exciting.

Luke refactored a bunch of code into js/src/vm/Stack.h. In a comment on the JS engine internals group, he wrote (emphasis mine):

David and Waldo raised the very reasonable question of whether js::StackFrame should be in its own file rather than in vm/Stack.h (its big). My reasoning for not wanting to is that FrameRegs + StackFrame + StackSegment + StackSpace + *FrameGuard altogether form a single logical data structure which I’d like to present as a whole [...] The same perspective shows up in math as many-sorted algebra so its not just C++ crazy-talk

None of us could figure out what that last sentence meant. We prevailed upon Luke to elaborate. I thought his explanation was a nice insight, so I’m sharing it here.

I’ll hazard an answer, knowing full well that there are at least three PhDs on the list who know a lot more about this than I do and may smite my answer with truth. (I’m at a layover in Hong Kong — be gentle

A single-sorted algebraic “structure” is something like a monoid, group, ring, field, etc: an abstract domain with a collection of operations (over this domain) and axioms that the operations must satisfy (e.g., distributivity, associativity, commutativity, etc). A single-sorted “algebra” implements a structure by picking a particular domain and set of operations that satisfy the axioms of the structure. (For example, the ring structure specifies an abstract + and * with a couple of axioms (associativity of +, distributivity, inverse for + and *, etc); the integers with arithmetic + and * are an algebra).

An important idea about all this is that, when you prove things about an algebraic structure, the proof is expressed only in terms of the declared operations/axioms of the structure, and not the particular details of any one algebra, so your theorem holds for all algebras of that structure. Now this starts to sound like abstract data types in computer science (s/structure/public interface/, s/axioms/specification/, s/algebra/concrete class/) and we can see that abstract algebraists are kinda like programmers who really really like reusable code.

A many-sorted structure/algebra is just the extension of the concept that can have more than one domain (thus, the operations can include more than one domain in their signature). An example is a vector space (which has a domain of scalars and a domain of vectors).

So then what’s the correspondence of these multi-sorted structures/algebras in programming? Classes/interfaces (of mainstream OOP languages) associate all operations with a single domain of values. I have little doubt there exist languages which solve the problem directly. We can hack multi-sorted-ness in C++ while maintaining some semblance of interface/implementation separation by just having multiple classes (one per domain) and making them all friends of each other. Then there is the question as to how to distribute the operations between the classes (or perhaps as non-member functions), but unless you want runtime polymorphism (which requires something like multi-methods), it’s a question of aesthetics.

I mentioned this originally because recognizing many-sorted algebras as a peer concept to single-sorted algebras helps to avoid a design mindset of “every class must be its own encapsulated island” which I feel can be detrimental when trying to modularize complex data structures like we have in SpiderMonkey. The Stack was one example; I think low-level objects + property tree will be another.

At the end of part 1, Cameron McCormack had just tracked bug 434494 down to two .focus() calls in browser.js. It was a race condition: whichever .focus() call happened to run last got to keep the focus.

You might have wondered why these calls were happening in a random order to begin with. As we’ll see, Cameron was able to fix the bug without investigating this! But I’ll talk a little bit about it because it’s interesting, and it can bite ordinary web pages, too.

Opening files is a very common cause of non-determinism. You call a function like window.open() or XMLHttpRequest().open to make it happen, and the call returns right away, even though the document hasn’t finished loading yet, so the application can remain responsive. The system will send a load event later, when it’s done. Because loading a file takes a small but somewhat random amount of time, the order of that load event compared to other events can be slightly random.

The root cause of bug 434494 is indeed the timing of load events. But Cameron found a way to fix the bug without having to track down the root cause, as we’ll see—after a little detour…

       <heycam> ok, so we know we have these two focus calls running in either
                order
   <jorendorff> hang on
   <jorendorff> I've got a Firefox 4 nightly on my Mac. So I suppose I have a
                browser.js in here somewhere.
   <jorendorff> One hesitates to encourage people to edit their browser app
       <heycam> ah yes, if you want to try it
       <heycam> heh
   <jorendorff> but just for my entertainment...
   <jorendorff> I can just throw an alert() in here?
       <heycam> I don't know how well it works to just edit the browser.js in
                whatever jar it lives in
       <heycam> it might work
       <heycam> I tended to just do a rebuild
   <jorendorff> you edited the source file?
       <heycam> yeah
       <heycam> then make, as usual
       <heycam> but probably editing browser.js in situ is fine
   <jorendorff> ...What about a chrome debugger? Ever use one of those?
       <heycam> no, well only just yesterday
   <jorendorff> heh
   <jorendorff> let's try editing it in place 
       <heycam> ok 
   <jorendorff>  /Applications/Nightly.app/Contents/MacOS/
   <jorendorff> omni.jar
       <heycam> I'm guessing so
   <jorendorff> chrome/browser/content/browser/browser.js
   <jorendorff> now to see if Emacs will let me edit this file in the middle
                of a jar
   <jorendorff> gosh, I think it will
       <heycam> would surprise me if it didn't 

If you’re on Mac or Linux, and you have Emacs, go ahead and find omni.jar in your Firefox installation directory, make yourself a nice backup copy, and poke around inside. If you like, you can make the exact change I made. Search the archive for browser.js, open it, then search for BrowserSearch_webSearch. Add an alert, like so:

Save it… and…

   <jorendorff> restarting... *fingers crossed*
   <jorendorff> hmm, no good.

It didn’t work. But are we discouraged? We are not. Later I asked about this on irc.mozilla.org, and I found out that you can force Firefox to read omni.jar again by passing the right command-line flag. First make sure Firefox is closed. Then:

  $ cd /Applications/Nightly.app/Contents/MacOS
  $ ./firefox --purgecaches

If you’re on Mac OS 10.5, you need to use a slightly different command to start Firefox:

  $ arch -i386 ./firefox-bin --purgecaches

And of course if you are using the official Firefox 4 instead of the unstable Nightly, you’ll find the file under /Applications/Firefox.app instead.

Then, when you hit Cmd+K:

You might want to put your original omni.jar back when you’re done. Software updates run a bit more smoothly when you haven’t been manually editing the programs.

Back to the bug:

   <jorendorff> so did you manage to convince yourself that this is what was
                happening by using alerts?
       <heycam> yes
       <heycam> alerting what the currently focused element was, at the time
                of the gURLBar.focus call
       <heycam> my thought was, not knowing anything about the random
                ordering, to ensure that the gURLBar isn't focused if we'd
                already done the search bar focusing
       <heycam> so I made an assumption, which I verified
       <heycam> that I could tell whether something had been focused already,
                by seeing if commandDispatcher.focusedElement was null
   <jorendorff> ah
       <heycam> alerting just before gURLBar.focus is called, in say a
                Cmd+N new window, showed me that focusedElement was null

This led to Cameron’s first patch for the problem. He wrapped the code that calls gURLBar.focus() in an if block, so that it only happens if we haven’t already focused something else. You can see those changes here. (The left side shows the code before the fix; the right side is how it looks after the fix. Incidentally you can find that page yourself by visiting bug 434494, then clicking “Show obsolete patches”, then clicking on the word “Diff” next to the first patch.)

   <jorendorff> the first part, i see the if statement you added
   <jorendorff> but what is that second change?
       <heycam> see the comment above the setTimeout 0 call
       <heycam> clearly it's trying to do something to avoid the url bar focus
   <jorendorff> oh
   <jorendorff> heh - yeah, it looks like someone thought they fixed this
                before. 
       <heycam> initially I only wrote the first hunk, and that worked.
   <jorendorff> That being done, the old setTimeout hack wasn't needed
                anymore, so you ripped it out.
       <heycam> exactly
       <heycam> there's no point in delaying the webSearch call in the
                setTimeout

The second half of the patch was just a little code cleanup.

Now Cameron had a working fix. He posted it in Bugzilla and requested a code review from Gavin Sharp, a long-time Firefox hacker who’s an expert in this area of the code. Gavin lives in Toronto.

Code reviews are interesting. You almost always learn something.

   <jorendorff> It looks like you spoke with gavin about this first patch
   <jorendorff> and you decided to make a few changes based on that.
       <heycam> yes
   <jorendorff> I don't suppose you have that chatlog...?
       <heycam> I do, hang on

Here is what Gavin said…

            <gavin> hm, that patch seems to be relying on the ordering of the load events
           <heycam> how's that?
            <gavin> and I'm not sure that it's safe to assume that focusedElement will be null on initial load
            <gavin> BrowserStartup runs off a load event
            <gavin> so does BrowserSearch.webSearch, with your change
           <heycam> I see
           <heycam> is there something I can listen for once the browser startup is done?
            <gavin> not at the moment
           <heycam> or maybe a way I could pass an argument into the browser
           <heycam> rather than listening for an event
            <gavin> oh, I lied
            <gavin> you could use browser-delayed-startup-finished
            <gavin> slightly messy because it's an observer notification, but doable
           <heycam> ok
           <heycam> I'll try that and post another patch. thanks!

Depending on .focusedElement was too iffy. Gavin suggested a different approach, which led Cameron to make a second version of the patch.

   <jorendorff> gavin left some new comments in the bug 
       <heycam> yes, I plan to get to them today 

Since my chat with Cameron last week, he posted a third version, which now bears Gavin’s seal of approval. It might be checked in by the time you read this. Sweet!

       <heycam> the one hiccup was the time between my posting the original
                patch, and getting comments from gavin
   <jorendorff> Was that due to the Firefox 4 crunch?
       <heycam> probably. gavin also seems to have a long request queue.
       <heycam> so a week ago I pinged him on email
   <jorendorff> ah

So I glossed over something: the three months between Cameron’s first patch and his chat with Gavin.

I hate to admit it, but I still have review requests from that era too. Firefox 4 was in beta, monopolizing developers’ time, for months. A lot of valuable work got put on hold. We’re never doing that again, thank goodness.

But sometimes a code review gets lost in the shuffle anyway. If yours takes more than a few days, it’s important to speak up:

       <heycam> although I'm usually hesitant to prod people
       <heycam> but that mail got swallowed 
       <heycam> so I pinged him on irc yesterday to ask him about it
       <heycam> in general I have no idea how long I should be letting my
                patches sit in someones queue before bugging them about it 
   <jorendorff> At least 3 days. But never 2 weeks.
       <heycam> I think that aspect of the process should be emphasised to new
                contributors
       <heycam> getting review, expectations of turnaround time
   <jorendorff> As long as you're super nice about it, a reminder is usually
                welcome, in my experience
       <heycam> ok

So this is what open source development is like. There is some stuff that looks like work, stuff that takes multiple tries, delays and waiting. There are also cool things to tinker with, puzzles waiting to be solved, useful skills to learn, and smart people around the world who would love to help you.

Don’t be shy.

This is a true story about fixing a bug.

In this story, we’ll see that a bunch of the Firefox UI is written in JavaScript. We’ll do something extremely inadvisable but sort of cool. And we’ll hit on one of the key life lessons of open source development.

Our hero is New Zealand-based Mozilla hacker Cameron McCormack. Cameron knew about this bug in the first place because someone (Marco De Vitis, to be specific) took the time to report it at bugzilla.mozilla.org. It’s bug 434494.

I chatted with Cameron about the bug and what happened next.

   <jorendorff> So this bug is about what happens when you have no windows open
                and you hit Cmd+K
       <heycam> right
   <jorendorff> which I guess only really happens on the mac, right?
       <heycam> yeah
       <heycam> where closing all the browser windows doesn't quit the app

It’s normal on the Mac for an application to stay open when its last window is closed. So in Firefox on the Mac, if you close all your tabs, Firefox is still there, and you can open a new browser window from the menu or by using a keyboard shortcut. Cmd+K is the shortcut for the search bar.

   <jorendorff> OK. So Cmd+K creates a new window in that case, and the bug
                was, sometimes it would focus the search box as desired,
                sometimes the location bar instead.

   <jorendorff> how did you track it down?
       <heycam> I *think* I searched for where the location bar got its focus,
                first

(Cameron doesn’t remember clearly because it was a month or two ago. More about that in part 2.)

       <heycam> ok
       <heycam> http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser.js#1485
       <heycam> so that's the only line that focuses the url bar
       <heycam> (it seems)
   <jorendorff> aha

I clicked the link and there was browser.js: nine thousand lines of JavaScript code that implement a lot of the UI around the edges of your browser window.

Well there you go. Focus the URL bar. There’s your bug, right?

But this can’t be the whole story, because in other circumstances, Cmd+K works. Somewhere there must be code for focusing the search bar that isn’t being called or isn’t working properly in this case. Cameron had no idea where that code might be.

So he went to mxr.mozilla.org and searched for it.

       <heycam> that's right
       <heycam> there's a file that has a whole bunch of keyboard mappings to
                actions
       <heycam> something.inc
       <heycam> browser-sets.inc
   <jorendorff> http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser-sets.inc
       <heycam> yeah
       <heycam> I still have no idea how all the xul things fit together
       <heycam> but having found this file, I could see that it associates
                Cmd+K with a "command"
       <heycam> Tools:Search
       <heycam> http://mxr.mozilla.org/mozilla-central/source/browser/base/content/browser-sets.inc#230

       <heycam> and then my thinking is "what defines commands"
       <heycam> so I'd grep that directory for Tools:Search
       <heycam> $ cd browser/base/content
       <heycam> $ grep Tools:Search *
       <heycam> browser-menubar.inc: command="Tools:Search"/>
       <heycam> browser-sets.inc:    <command id="Tools:Search" oncommand="BrowserSearch.webSearch();"/>
       <heycam> browser-sets.inc:    <key id="key_search" key="&searchFocus.commandkey;" command="Tools:Search" modifiers="accel"/>
       <heycam> browser-sets.inc:    <key id="key_search2" key="&findOnCmd.commandkey;" command="Tools:Search" modifiers="accel,alt"/>
       <heycam> browser-sets.inc:    <key id="key_search2" key="&searchFocusUnix.commandkey;" command="Tools:Search" modifiers="accel"/>
       <heycam> that <command> element looks right
       <heycam> that gets us to the webSearch function in browser.js

Cameron saw oncommand="BrowserSearch.webSearch()", guessed correctly that it was a snippet of JavaScript, and with a bit more searching, tracked down the webSearch function. It also lives in browser.js, it turns out.

   <jorendorff> so now we know two things
   <jorendorff> We know where focusing on the search bar is supposed to happen
                (and roughly what pointers the system is chasing to get from
                Cmd+K to this JS function)
   <jorendorff> we also know there's some other code in browser.js that
                focuses on the url bar.
       <heycam> right
   <jorendorff> and the hunch is then "i bet both are happening"?
       <heycam> both are happening, but in an order that's not fixed
   <jorendorff> now what?
       <heycam> I may have put some debugging code in to make sure that both
                are running, not sure
       <heycam> like an alert or something
   <jorendorff> That was the next thing I was going to ask about.
       <heycam> without knowing how chrome js works, it's hard to know how to
                even do simple printf debugging
   <jorendorff> Of course anybody reading this--certainly if they happen to be
                running Firefox 4 on mac -- would like to watch this happening
       <heycam> yeah

Would you? Maybe we’ll do that in part 2.

       <heycam> will you be making a guided tutorial to fixing this bug?
   <jorendorff> Weelll, It depends on how much time I end up with :-\
       <heycam> I think UI stuff is a great place for new people to start
                working on, because it's very immediate
       <heycam> and you can see if things are working or not
   <jorendorff> i agree!
       <heycam> but, there's an awful lot of complexity in there
   <jorendorff> well, that's how it is
       <heycam> undocumented too 
   <jorendorff> but what you're showing people is how to ignore things rather
                than get overwhelmed
   <jorendorff> this is valuable
       <heycam> yeah, I think that's the key
       <heycam> ignoring things, plowing on
       <heycam> and if you miss important things, others will let you know
       <heycam> that's how I try to approach areas I'm unfamiliar with
       <heycam> I'll have a bash at it, post it for review, and learn what
                things I didn't know from review comments 
   <jorendorff> as in this case 
       <heycam> yeah
   <jorendorff> but that gets ahead of the story

We’ll get there in part 2.

Now about that life lesson.

You probably noticed Cameron has been tracking down the problem by randomly searching the code for words that might be relevant. This might not seem worth documenting. In fact it may seem a little embarrassing—you mean he didn’t know the code well enough to figure out where the bug was using only his enormous brain? But this is exactly what I think is most important about this story. What Cameron is doing requires some programming chops, and it requires the ability to navigate undocumented monster-infested waters—but it’s not magic.

You could do this.

Maybe you are not an expert in everything. Maybe (like me) you’re a shy person working in a complicated world with incomplete knowledge. How can you function like that? How do you know if you have anything to contribute? How can you get involved in new things?

Step 1: Don’t panic.

Step 2:

• search for the answer
• try something and see what happens
• ask someone

To be continued.

Update: Part 2 is up.

I got tagged to tell you seven things about myself back in January, and I finally found a seventh, mildly interesting thing to write. So here you go.

The rules

1. Link back to your original tagger and list the rules in your post.
2. Share seven facts about yourself.
3. Tag some (seven?) people by leaving names and links to their blogs.
4. Let them know they’ve been tagged.

Seven things

1. I’m married and have three kids. All our noses make a different noise when squeezed, except that mine goes “AWOOOGA” and the baby’s goes “awooga”. He has my nose.
2. I love math. I have a shirt with Łukasiewicz’s axioms of propositional calculus on it. (Incidentally, if you take Pascal’s triangle and color the even numbers black, you get Sierpinski’s gasket. Who knew?)
3. My memory for appointments, names, and faces is amazingly bad. I also have an innate incompetence for using calendaring software. I somehow manage to convince it not to alert me about a meeting; or I miss the alerts; or they go off at maximum volume but only in the middle of the night or when I’m at the store; or I see the alerts but can’t decipher what I wrote; or my calendar gets so full of obsolete recurring appointments that I ignore all the alerts (which is the steady state and current situation).
4. I am such a slow reader that I rarely finish a book. This is especially true of nonfiction books (I like to read “hard” nonfiction, like textbooks). There’s no big payoff ahead, pulling me toward the end. At some point it seems more worthwhile to start reading some other book (which I also won’t finish).
5. I used to really like table-top games, like board games, back when I had time to play them. Now I only like games that are actually fun, which is a different scene. My favorites include 1000 Blank White Cards and a roleplaying game named EARS, about which details are available on request.
6. I love making up stories and telling them to my kids. I keep thinking I’ll write them down someday. I probably won’t. But I write down the outlines, hoping I can piece them back together later.
7. I live in the country about half an hour outside of Nashville, Tennessee. There’s a room in my house where you can watch the sunrise and the sunset. We’ve seen deer, rabbits, and a whole family of wild turkeys in our yard, to say nothing of lizards, turtles, and birds. In spring, the birds get so loud around eight in the morning that it’s hard to work. On summer evenings there are hundreds of fireflies.

If you want to know more about me, you could read my other blog. Why do I have two? I don’t know.

I can’t bring myself to tag anybody, since this took so many months of my time. But if you’d like to play, and somehow missed all the action back in January, do feel free.

I use Mercurial Queues. Sometimes I make a bunch of changes intended for one patch when in fact the currently applied patch is something else. If I manage to run hg qrefresh before detecting the mistake, it munges my new work with the unrelated patch.  The damage is a huge pain to undo.

One day we were all complaining about this on IRC, and Chris Jones wrote this line of code (at Graydon’s suggestion):

PS1='\u@\h:\w[\[`hg qtop 2>/dev/null`\]]\$ '

Put it in your .bashrc and your bash prompt will look like this:

cjones@hell:~/porky[bug-545432-newexprs]$

Check that out, qtop in your prompt!  The idea is to stop you before you hit enter.  Maybe it’ll work for you, maybe not.

I use something a bit more complicated, but it amounts to the same thing:

function mercurial-qtop() {
    qtop=`hg qtop 2>/dev/null`
    if [ "$qtop" != "" ]; then
        echo " (+$qtop)"
    fi
}

PROMPT_COMMAND='MERCURIAL_QTOP=`mercurial-qtop`'
PS1='\[\e[1m\]\w$MERCURIAL_QTOP\$\[\e[0m\] '

Update: Changed my code snippet to use PROMPT_COMMAND. bash is amazingly bad at guessing how long your prompt is, even though it knows PS1. This seems to make readline do bizarre things when you cut and paste, resize the window, or exceed a single line of input.

This summer, thirty interns participated in the Mozilla summer internship program. They worked in virtually every area of the Mozilla project: Firefox, Thunderbird, testing, the build system, marketing, analytics, web development, IT. You can read their blogs here.

If you’re a college student, you can apply now for a Mozilla internship in summer 2009.

Sound good? Keep in mind that:

• you will have to spend about twelve weeks in sunny Mountain View, California.

• you’ll be surrounded by some of the most brilliant minds in Open Source software.

• you could have a lasting impact on the way over a hundred million people use the web.

If you think you can live with all that, apply online (that web site is kind of weird; poke around for “Intern” positions) or send email to julie at mozilla dot com.  Internships will be awarded by the end of February 2009 or so.