Here’s a quick transcript of the video:

Hello, this is Rob Helmer from the Mozilla WebDev team.
I’m here to introduce a project we’ve been working on
for the past few months: Perf-O-Matic 2.0

Perf-O-Matic is the web application we use to display Firefox performance
data. Every time a developer “checks in” a change to Firefox, a fleet of
machines runs the browser through a battery of tests to measure performance.

These tests range from startup/shutdown performance, the time it takes to
load a web page, and browser benchmark suites like WebKit’s SunSpider,
Mozilla’s Kraken, and Google’s V8 Benchmark.

Perf-O-Matic 2.0 features a new, streamlined interface, designed by Chris Howse.

This is the dashboard, which allows for a quick overview of popular tests.

Clicking on any graph in the dashboard brings us to the
“Custom Chart” screen. The graph is interactive – hover over any point
to get a tooltip with more information about this test result. Every
test result will have an associate changeset id in mercurial,
which links to more information about what changes the developer made.

The toolbar of the “Custom Chart” screen allows us to select date range,
data type (running time, time delta, time delta %), zoom in and out,
and you can even show all changesets for a highlighted range.

The “Link” button gives a quick link to the chart you’re looking at.
“Embed” gives you some HTML code you can use if you want to display
interactive graphs on your own site. I’ll give a quick demo of this.
Here’s an embedded, interactive chart; this is inspired by arewefastyet.com.

On the left, we have information about the currently loaded
data series – SunSpider benchmark for Firefox on Windows 5.1. Each point on
the graph is actually an average of several test machine runs, so we can
“explode” the data series to see the raw, individual data. Clicking again
will “implode” the data, showing us the more human-readable average.

Finally, more test data can be added, for making direct visual comparisons
between multiple data series. You can multi-select here, to get multiple data series at once.

This project builds on the work of many others, including (but not limited to):
vlad
rdoherty
anode
ianbicking
many, many others on the graphserver side

We also received valuable early design review and feedback from jonath, rsayre, eshan, and many others.

Perf-O-Matic uses many existing open-source projects, such as python, flot, and node.js

Perf-O-Matic 2.0 is being staged for testing.
We’ve got a handful of known bugs, and we need your help to find the rest.

Please try it out at:
http://graphs-stage2.mozilla.org

You can find more information on the Perf-O-Matic UI wiki page, including links to mercurial and github:
https://wiki.mozilla.org/Perfomatic:UI

1. eval’ing JSON in the browser is unsafe. Using native JSON parsing protects you against possible code execution.
2. Safely eval’ing JSON with a 3rd party library can be orders of magnitude slower. Native JSON parsing is much faster.

How does native JSON work compared to plain old eval? Simple:

var jsonString = '{"name":"Ryan", "address":"Mountain View, CA"}';
var person = JSON.parse(jsonString);
// 'person' is now a JavaScript object with 2 properties; name and address

Pretty easy huh? And here’s how to get a JSON string from an object:

var personString = JSON.stringify(person);
// 'personString' now holds the string '{"name":"Ryan", "address":"Mountain View, CA"}'

“But wait!”, you say. “How is it safer? How much faster is it compared to eval?”. Ok, I’ll show you.

Native JSON parsing in Firefox 3.1 is safer because it does not support objects with functions. Attempting to convert an object with functions into a JSON string will only convert its properties and not its functions. And any malformed JSON string will result in a parse error instead of possible code execution.

Now, regarding speed, native JSON parsing is faster, much faster. Instead of pretty charts and graphs, I’ll give you a real-world example.

The new Graph Server uses a JSON API to fetch test information and results, so I figured it would be a good application to benchmark. So I wrapped our code that parses the JSON response with some Firebug profiler calls:

    console.time('parsejson');
    var obj = eval('(' + resp + ')');
    console.timeEnd('parsejson');

Loading a test’s results (array with 3,000 indexes, 24k gzipped) gave me a time of 125ms. (Repeated tests yielded results +/- 5ms). Then I changed eval to JSON.parse:

    console.time('parsejson');
    var obj = JSON.parse(resp);
    console.timeEnd('parsejson');

Which resulted in an average time of 40ms! That’s about 2.7 times faster with 1 line of code changed. Not bad!

Granted, a difference of 80ms isn’t that much, but in an AJAX (or, more accurately, AJAJ?) application, it can add up.

What’s the use of native JSON if it’s only available in Firefox? Luckily, IE8 has implemented it in RC1, which is rumored to be released in March. Hopefully other browsers will follow suit too, but for now it’s best to use a JSON parser such as the one on json.org. It’s small, safe and will not override native JSON implementation if detected.

Points to remember:

• Plain old eval is unsafe (especially if you don’t trust the source), use a JSON library to protect yourself.
• Use native JSON when available.
• Bug other browser vendors to implement native JSON

For those that don’t know, the Graph Server is used to display performance test data of Firefox builds reported by Talos.

Our work initially started as performance improvements and some new features, but the more we worked with the old architecture, it became quite apparent it would not scale (performance and feature-wise).

The old database schema duplicated test data in multiple tables and stored similar, but different data in the same tables. Tables had ballooned to millions (and billions) of rows that were queried for basic information such as all unique test names, resulting in queries that ran forever. And the queries that did finish were looped over in JavaScript to pull out test information, resulting in the browser locking up because it was looping over hundreds of thousands of rows.

If it’s not clear already, one of main issues was with the database schema; it needed to be normalized.

Here’s the old, non-normalized schema:

And here’s the new, normalized schema after the team was locked in a room for an afternoon:

Much cleaner, no duplicated data, easy to understand the various machines, branches and tests that are used for displaying test data. No need look at entire tables to find basic information such as test names.

With this new schema in place, it also required a re-write of our server-side scripts we use to fetch test information for the front end graphing component. Since Mozilla is as open as possible, instead of just changing what was needed, I decided to implement a JSON API that would allow anyone to easily retrieve test data.

Lastly, our Talos <--> Graph Server communication needed to be re-written. Lars rewrote the collector script that accepts values from Talos and Alice rewrote the pieces of Talos that send data to the Graph Server.

After all that work, we now have a working stage server (Firefox 3.1 or higher required due to native JSON requirement) with our new code. We have a bit more testing and some performance benchmarking to do before it goes live, but we’re happy that all the pieces are working.

Want to know more? We have a wiki page with more information at https://wiki.mozilla.org/Perfomatic#Rearchitecture.