Biggest snag so far is figuring out how to either do nsNavHistoryContainerResultNode::FillStats’s work inside the database, rather than C++, or avoid the work altogether. FillStats recursively calculates a couple of members of a node and its descendants, and these members are then used to subsequently sort the children, if the selected sort requires those members.

Also filed a couple of bugs for tracking work on 3.7’s new Places UI:

• Bug 529597: Places protocol handler
• Bug 529605: Places data should be visible in content

Wanted to work on the former this week but just didn’t get around to it.

And all the while I wonder, do people still use bookmarks in Firefox?

Created a new project, async Places containers, which is the topic of the remainder of this post.

Spent time thinking about a Storage “live view” interface, mocked it up, talked to sdwilsh about it. The idea was a live view, where modifications to the view’s underlying data would automatically trigger a view refresh. The motivation was the fact that Places query results now (well, will—bug 499985) use the same collations in two locations: in the database on initialization (which makes possible APIs that return data in async batches), and in code when changes occur to the results. That means duplicate collation code in Storage and Places, and the collations used by each must be equivalent. But if there were such a thing as a generalized live view, it could all be handled by Storage.

However, I realized it’s not possible to in general and in constant time determine whether an arbitrary change to the database occurred within a given view. In the current code it’s O(1) to do this check against a list of query results exactly because a list of query results is not generalized. So I dropped this idea.

The problem of duplicate collations remains, and now I’m thinking of creating temp tables in Places code to store query results instead of the arrays of nodes it currently uses. That way Storage and its collations do the heavy lifting, handling both initialization of and updates to results. We’d turn the data into nodes only when it’s leaving the API. There are big roadblocks to doing that, though. There’s lots of post-Storage processing and filtering of data that would need to be moved to Storage. I think the reasons we don’t do this processing in Storage have more to do with an inadequate schema and layers of modifications to the code over a long period of time than any substantive problems with using SQL or Storage. In general I really wish we took better advantage of SQLite.

Rather than trying to solve big problems right off the bat, it might be better to start small: forget about the duplicate collations for now (making sure they’re equivalent) and just update the initialization path of the bookmarks query results, the simplest kind, to use the Storage collations. Even that isn’t so straightforward. We have a single, immutable mozIStorageStatement—among many others—that gets a container’s children. It’s created from a long string of raw SQL. If I merely want to change the sort order of that query, or any other for that matter, it’s not so clean or easy.

Jetpack is a platform for writing Firefox add-ons the same way you write for the Web: HTML, CSS, JavaScript, Webby libraries like jQuery, no browser restarts to see your changes. But that’s shortchanging it. It’s also add-ons with a security model; an interactive shell for tinkering with the browser, since it bundles Bespin; a playground for budding coders, potentially; portable to other XULRunner apps like Thunderbird; and a mechanism for extending not only the browser but the Web itself. If it’s easy now to dismiss Jetpack as add-ons lite, it’s only because it’s still teething.

The new menu API lets you manipulate the browser’s menus—not only menu bar and context menus, but menus you can create and attach most anywhere, chrome or content, popup or context.

These lines of JavaScript add a “Tweet This Link” item to the page’s hyperlink context menu:

jetpack.menu.context.page.on("a").add(function (context) ({
  label: "Tweet This Link",
  command: function () jetpack.lib.twitter.statuses.update({
    status: context.node.href
  })
}));

Try it out.

This one adds a “Recent Tweets” submenu to the Tools menu. When it’s opened, it displays a “Loading…” item, and if it remains open when the data comes down from Twitter, it’s filled in with solipsism, one item per tweet. Note that the word “Tools” doesn’t appear, nor does any position specifying where in the menu to insert the item. You tell Jetpack, “This is my add-on’s menu, stick it in the best place.” You can get specific if you want.

jetpack.menu.add({
  label: "Recent Tweets",
  menu: new jetpack.Menu({
    beforeShow: function (menu) {
      menu.set("Loading...");
      jetpack.lib.twitter.statuses.public_timeline({
        success: function (array) {
          let tweetArray = array.map(function (obj) obj.text);
          menu.set(tweetArray);
        }
      });
    }
  })
});

Try it out. (Mac users, a caveat.)

You can add popup menus to content in pages. This adds a popup to all images on the Web. Left-click any image in a page and you’ll get a menu that notifies you of its URL:

jetpack.tabs.onReady(function () {
  $("img", this.contentDocument).each(function () {
    var imgNode = this;
    var myMenu = new jetpack.Menu([{
      label: "Show Image URL",
      command: function () jetpack.notifications.show(imgNode.src)
    }]);
    myMenu.popupOn(imgNode);
  });
});

Tile and grout.

The first two snippets also illustrate Jetpack’s new Twitter API, which hews close to Twitter’s own. If you’re already familiar with that API, there’s nothing new to learn, and if you aren’t, it’s easy, and you can add the phrase “Twitter API” to your resume. (You’re welcome.)

If you have thoughts on these new APIs—or bugs or patches—let us know. More info and examples at the links below.

• Menu API and examples
• Twitter API and examples
• Jetpack home
• Find a bug?
• Discussion list

• Started looking again at making Places query APIs async, bug 490714. First need to tackle bug 499985, sorting of query results should be done entirely in SQL.
• “Helped” bz with test failures in blocker bug 526178, binding loading order changed in 3.6b1 compared to previous version of firefox.
• Awaiting Josh’s review on bug 506814, get rid of / Change GetPersistentDescriptor / SetPersistentDescriptor. Spoke to him last week, think he’s pretty busy with more important things.

→

• Work on bug 499985.
• Been thinking about the gap between the Places database and its query code and the janky structure that bridges it. Think more about it, with intensity.

Shiretoko (3.5)   10266.25 ms
Namoroka  (3.6)   10017.55
Minefield (trunk) 11023.65

If we had had cold Ts when we landed whatever caused Minefield’s 10% regression over Namoroka, we would have caught it.

Cold Ts is Ts run under simulated cold startup. Right now we’re able to simulate cold startup on OS X only, so the test is not yet switched on for Windows and Linux. The procedure on Linux, echo 3 > /proc/sys/vm/drop_caches, requires root privileges, which either opens a security hole on the Tinderboxen or forces a sudo prompt. Windows is worse: Nothing we know of works well or at all except for reboot, and the Talos infrastructure makes rebooting impractical. Any help here would be greatly appreciated.

• Bug 510587 – Create a cold startup Ts
• Bug 511914 – Allow Talos tests to run head and tail scripts
• Simulated cold startup notes

Good news is, joelr joined the platform team and is kicking startup in the shins. He’s on the ball, I hope he doesn’t mind my saying.

My OS X-specific XPT-linking patch from last time landed. Nightly trunk (Minefield) users on OS X, I’d like to hear from you if you notice an improvement in cold startup time. I was unable to, however, despite noting a good decrease in I/O. Ted is working on improving OS X packaging overall in bug 463605.

Making progress on setting up a Talos cold Ts. A byproduct is that we’re going to allow Talos tests to run head and tail scripts, which may be useful for others. Simulating a cold startup on Windows looks like it might be a little tricky, but other than that I don’t foresee any roadblocks. (What could go wrong?)

Posted a list of all files touched on OS X cold startup, including time spent in the associated syscalls, number of bytes touched, and throughput. You can drill down into each file to see all the I/O-related syscalls on it. This is data from one run only, so it’s not smoothed out across multiple runs. I have noticed some jitter between runs: For example, reading four bytes of search.json will take 200ms (!) during one run, but on most others it’s something like 10-20ms (!). The order of files by duration tends to remain fairly consistent, though. I’ll try to keep this up-to-date; already it’s several days old, and since then I’ve improved the output so that it shows the counts of each syscall on the main page. Feel free to rummage through my startup dump.

My daily startup notes, data, and scripts are open as always. For details and context, check there.

Yesterday I filed bug 510309 to combine XPT files for OS X DMG packages. It should shave a couple hundred milliseconds off of OS X cold startup time spent in I/O. Dietrich, Alice, and I are setting up a Talos Ts to measure cold startup, and that will provide more confidence in this number and more accurate and stable numbers in general.

Update: Jesse pointed out that percentages are helpful when talking about performance. I’ve been getting 8 to 9 wall clock seconds pretty consistently on OS X (simulated) cold startup. Taking the 200ms decrease in I/O (not wall clock time), which is toward the low end of some of the runs I’ve done, that’s a 2.5% improvement over 8s. In the bug linked above I note a decrease of 285ms, a 3.6% improvement. One big caveat is that I was unable to see a corresponding drop in wall clock time in my testing, which is one reason I’d like to get a Talos cold Ts set up.

SQLite provides a few simple built-in collations: BINARY, NOCASE, and RTRIM, but as the first suggests they all use memcmp and ignore text encoding and the user’s locale. If you want to show respect to your user and the vagaries of her language’s collating conventions, you have to load your entire set of results and then sort it manually. Sweet.

But now you can do it all in SQL. Bug 499990 adds the following collations sensitive to the locale of your user’s application:

locale

Case- and accent-insensitive

locale_case_sensitive

Case-sensitive, accent-insensitive

locale_accent_sensitive

Case-insensitive, accent-sensitive

locale_case_accent_sensitive

Case- and accent-sensitive

That’s everything covered by the platform’s existing collation facilities.

Use them like so:

SELECT * FROM fooflefipples ORDER BY name COLLATE locale ASC;

Locale-sensitive collations are useful for everyone building on the platform, but I added them as part of some ongoing work on async Places APIs in Firefox. We’d like to notify consumers as batches of results load from the database, but that’s not possible if we have to sort the entire set outside the database — which in turn increases Places’s code size for something that should be handled at the platform level, and now it is.

For a nice summary of the importance and difficulty of sorting strings in our multilingual digital world, see the introduction to the Unicode Collation Algorithm specification.

In related Storage news, Curtis is adding a Levenshtein distance function. (Pretty cool, but my patch is way cooler, don’t tell anybody.)

For example, if you have lots of bookmarks or history, you might have noticed that searching in the history sidebar or bookmarks library can be a little, uh, unresponsive. When you search or click a tag, folder, or history container in one of the sidebars or library, Firefox drops everything to fetch your results. I’ve been working on making these actions asynchronous so that Firefox can walk and chew gum.

Try out a test build here. It’s based off of Minefield, the trunk version of Firefox. Warning: You can use this with your default Firefox profile, but only if you normally use Firefox 3.0, the current 3.5 nightly, the current Minefield nightly, or 3.5 RC 1. (As always, however, it’s a good idea to backup your profile before you try any experimental build.) If you run any of the 3.5 betas up to and including beta 4, you will need to make a copy of your profile and use it instead.

This build loads searches and tag, history, and query containers asynchronously. Instead of becoming unresponsive as it loads the entire set of results, Firefox will display results batch by batch as it retrieves them. That’s the plan anyway. Open up the sidebars and library and give it a shot.

A few notes about what not to expect:

• Responsiveness is still not 100%. Depending on the size of your history, searching and opening large containers remain a little jerky, especially the first time. Many disparate pieces impact responsiveness, and this one patch doesn’t touch them all. I hope you will notice some improvement, though, and I’d certainly like to hear if it makes things worse.
• This is not a cure-all. It addresses only the issues described in this post, not things like the awesomebar or deleting large numbers of bookmarks.
• Completeness and visual polish.

Follow along in bug 490714.

The Places Stats Project has been up and running for nearly three months now, and I previously blahhged about it and our initial results here. Everyone’s been busy getting Firefox 3.5 out the door since then, but now that work on it is winding down I’ll post soon about some of the things we learned.