At our recent Mozilla All Hands, I shared some slides about Shipyard, a JavaScript MVC framework that is making it’s way into Add-on Builder. It’s not finished, but since I shared it there, it felt appropriate to share what there currently is here.

Summary

Shipyard is an application framework that covers all the common things any JavaScript application would need to deal with: interacting with a server, storing data, rendering said data in the browser, and responding to user actions. An application built on Shipyard should only have to write the parts to pull all those things together.

If you’re application is going to have 1000 lines of JavaScript, would you rather write all those yourself, or have 900 of them be in a framework that is tested and used by others?

When starting a web application, you would reach for Django, or CakePHP, or Rails; never would you decide to use just the language itself. Why shouldn’t you do the same when the target language is JavaScript?

Framework-wide Goals

1. Be able to declare dependencies inside the code, and not be bothered with managing them during development or deployment.
2. Be easily testable, using a node test runner.
3. Not reliant on any other language. Build scripts will use JavaScript. The End.

More

It’s heavily influenced by MooTools, since they have an excellent modular design, but turned into CommonJS modules while Moo 2.0 figures itself out. There’s slides, and the repository, again. You could even discuss what true Controllers should do.

Installing Node.js on systems such as Linux or Mac OSX is straight forward and simple but, installing it under Windows is a little more tricky, that is until the port of Node.js to Windows is complete, allowing Node.js to run natively on Windows.

Your first step is to download and install Cygwin. For Nodejs you can grab the zip archive (remember, even numbers = stable releases, uneven numbers != stable) or you can clone the repository directly from Github. I am using the clone route for this article.

Getting Cygwin Ready

Whether you are going to work from a clone using Git or not, we need to step back for a little bit and add a couple of additions to Cygwin before we can continue. To install extra features run the setup.exe file you downloaded when initially installing Cygwin. Once you get to the select packages window, type git into the search input.

From the result, open up the develop tree and select git and git-completion(optional):

Next type in python, open up the interpreters tree and select the Python interpreter:

Next we need to add a C++ compiler, type g++ into the search input, open up the develop tree and select gcc-g++

We also need to add the openssl-develop package. Type in openssl, expand the Develop tree and select openssl-develop

Lastly, we need the all important make utility. Type make into search, expand the Develop tree and select make from the list:

Accept the required packages and click next again. Once the installation is complete, we can continue.

Getting The Nodejs Source

Open up Cygwin again and navigate to the directory you want to clone Nodejs to:

cd /cygdrive/c/github/

Now run the following command to clone the repo:

git clone https://github.com/joyent/node.git nodejs

After this is complete, move into the nodejs directory. We now need to checkout the version we want to build so, instead of just a plain checkout, we will specify the version we want:

git checkout v0.4.12

NOTE: If you have AVG AntiVirus installed it is going to mark the above process as a possible virus and prevent the checkout from happening successfully. Head into the AVG preferences and under ‘Resident Shield’ add an exception that points to the path where you Nodejs source will be located.

Building Node

Once the checkout is complete, run:

./configure

NOTE: If when running the above command you get an error stating “unable to remap…….”, you will need to rebase. To do this, close your current Cygwin window and open the Windows command prompt.

From here navigate to the bin directory inside Cygwin, for example

c:/cygwin/bin

From here run ash.exe. At the resulting bash, type /bin/rebaseall

Now, open up your Cygwin shell again and rerun configure. Once completed, run make and lastly run make install. Once this is DONE, you have a Nodejs environment on Windows.

Testing Our Installation

To test that our environment does indeed work, let’s use the staple Hello World example from the Nodejs site. In a folder anywhere on your computer, create a new file and call it hello.js, inside this, add the following:

var http = require('http');
  http.createServer(function (req, res) {
      res.writeHead(200, {'Content-Type': 'text/plain'});
      res.end('Hello World\n');
  }).listen(1337, "127.0.0.1");

console.log('Server running at http://127.0.0.1:1337/');

Now, navigate to this file location using Cygwin and run the following command:

node hello.js

If you get a response such as “Server running at http://127.0.0.1:1337/”, Nodejs is working. For more proof, open up that URL and stare in amazement. And that is it, a working Nodejs installation on your Windows machine, enjoy!

I will be following this article up with a short one on setting up NPM, the Nodejs package manager, on Windows. On a side note, there is currently a native node.exe installer for Windows but, this is in the unstable 0.5.8 branch so, although I have been told it should be fine for testing and local development, if you wanna live on the edge, give it a go.

• Groundwork for a plugin directory
• Cross browser plugin checking

The Backend

Les Orchard has created a backend to the plugin finder service. We’ve added another input to the call named ‘detection’ which will allow us more flexibility in how we match known releases to OS / Product / Version / Plugin / Plugin Version combos. More news at 11, but he’s built the core pieces for a self-service plugin release application.

The Frontend

We updated the JavaScript client to support ‘modern browsers’ as well as IE.

But IE 8 is a modern browser!

Hmm, well it doesn’t have a navigator.plugins object. Other popular and recently released desktop browsers *do* have this feature. Heck, even some phone’s browsers have it.

Breaking News: The platform preview of IE 9 has a working navigator.plugins object! So IE 9 fits the modern browser category… Congrats to the IE 9 team! We’ll make sure the page works by the time IE 9 ships, filed Bug#566003.

Cross Browser Flavors

Our plugin detection uses one of three strategies:

1. Strategy 1) Iterate the plugins objects and parse a version string out of the description or name
2. Strategy 2) Iterate the plugins objects and use the ‘version’ property
3. Strategy 3) Instantiate well-known plugins and test their version via the pinlady.net version detection library.

If your goal is to protect as many users from as many known plugin vulnerabilities as possible… Strategy 3 doesn’t scale. Strategies 1 and 2 are dynamic and (in the best case) plugin agnostic. As new plugins come onto the market, the plugin finder service has to be updated, but no new code has to be written and shipped.

This is why IE plugin detection is limited.

Strategy 2 is the cleanest… and only supported by Firefox 3.6+. We would be pleased as punch if other browser vendors would create a version property. The Plugin Check page and other pieces of code that do plugin detection, will become more accurate.

We’re really excited about supporting all browsers and that is what Strategy 1 buys us. When a vendor has put a useful version number in the description or name, then it’s possible for our page to help Safari, Opera, or Chrome users understand their plugin versions better.

Geeky Aside:

Fly in the ointment, even for Firefox 3.6+ we currently will use methods #1 and #2 depending on what’s best for detecting the most accurate version for the most popular plugins. Why is nothing every simple?

What can browser vendors do?

Please implement the navigator.plugins[x].version property. This exposes an explicit plugin version number from the vendor.

Why?

It will keep your users safer. This and other security tools can detect vulnerable versions easier and more accurately.

What can plugin vendors do?

At a minimum, please put your full version number in the plugin description field. Also make this as exact as possible, include the build number etc. 1.1.2.9282 is better than 1.1.2. Bonus points, expose your version numbers in the version property, even on Linux builds of your plugin.

Why?

• Keep your users safe
• Improve your lastest version uptake
• Keep users coming back to your distribution channel
• Reduce your support costs

What’s next?

We’ve built the plumbing and populated it with some popular plugins versions. Our next major release will be focused on building a self-service plugin release management app, so that vendors can populate the data for the backend API.

Why is this important to you?

• Crashes are the number one concern for Firefox users, and we are listening.
• At least 30% of all Firefox crashes are caused by third-party plugins.
• Many major security vulnerabilities exploit out of date plugins.

Why is this important to Mozilla?

Increasing awareness about plugins makes the web better, and that’s our mission.

• We want the web to be safer.
• We want the web to be less crashy.
• We want to help everyone — not just Firefox users — to address the plugin problem. (though admittedly it doesn’t fully work with all browsers yet, it will)

What did we do?

The plugin checker has three components:

• The Server: Plugin Finder Service (PFS2)
• The Javascript: Perfides
• The Web Page: mozilla.com

The end result is actually pretty simple — and that’s how it needs to be. Here’s your plugins, and here’s their statuses:

Putting it all together, we reach a workflow similar to the graph below. Our goal is to query a central database that contains plugin information and inform users about the status of their plugins. This was built so it could be used to support Firefox directly in the future.

What will happen next?

The three components above are a good start, but together we can do more.

• Integrate this experience with the Firefox client. Firefox will identify vulnerable plugins and help you update them.
• Create a self-service panel for vendors to update their plugin info as new releases come out.
• Create an open directory for all plugin information (sort of like Plugindoc but dynamic)
• Evangelize plugin detection via an embeddable widget — get it out on WordPress, etc.
• Integrate with our crash reporting system so we have a report card/dashboard for which plugins are most crashy

How can you help?

This entire project is open source. You can work on any of these components to help contribute to the effort:

• View the server code for PFS2
• View the client code for Perfides
• File a bug if you find one
• Tell us about plugins we don’t know about

For those that don’t know, the Graph Server is used to display performance test data of Firefox builds reported by Talos.

Our work initially started as performance improvements and some new features, but the more we worked with the old architecture, it became quite apparent it would not scale (performance and feature-wise).

The old database schema duplicated test data in multiple tables and stored similar, but different data in the same tables. Tables had ballooned to millions (and billions) of rows that were queried for basic information such as all unique test names, resulting in queries that ran forever. And the queries that did finish were looped over in JavaScript to pull out test information, resulting in the browser locking up because it was looping over hundreds of thousands of rows.

If it’s not clear already, one of main issues was with the database schema; it needed to be normalized.

Here’s the old, non-normalized schema:

And here’s the new, normalized schema after the team was locked in a room for an afternoon:

Much cleaner, no duplicated data, easy to understand the various machines, branches and tests that are used for displaying test data. No need look at entire tables to find basic information such as test names.

With this new schema in place, it also required a re-write of our server-side scripts we use to fetch test information for the front end graphing component. Since Mozilla is as open as possible, instead of just changing what was needed, I decided to implement a JSON API that would allow anyone to easily retrieve test data.

Lastly, our Talos <--> Graph Server communication needed to be re-written. Lars rewrote the collector script that accepts values from Talos and Alice rewrote the pieces of Talos that send data to the Graph Server.

After all that work, we now have a working stage server (Firefox 3.1 or higher required due to native JSON requirement) with our new code. We have a bit more testing and some performance benchmarking to do before it goes live, but we’re happy that all the pieces are working.

Want to know more? We have a wiki page with more information at https://wiki.mozilla.org/Perfomatic#Rearchitecture.

To summarize the issue, for no apparent reason in right-to-left languages a really long scrollbar would appear at the bottom of the window.

Even though there was a scrollbar, when you scrolled all the way to the left, nothing was there. Another reason this was odd was the scrollbar only appeared in right-to-left (RTL) languages. Inspecting the page via Firebug didn’t give any clues as to what was causing the issue as there was no element hidden somewhere onscreen. Finally, to make things even weirder, the scrollbar only appeared when JavaScript was turned on.

After some thought, I had a feeling that we might be using absolute positioning to position an element to the left and above the page offscreen, which is quite common. In a RTL page, however, left is does not move an element outside a page’s boundaries. So the result is you get a scrollbar.

So what’s a web developer to do? Firebug to the rescue! I popped it open and started typing some JavaScript into the console to find an element that seemed really far offscreen:

var nodes = document.getElementsByTagName("*");

for(var i=0; i < nodes.length;i++) {
    var node = nodes[i];
    if(node.offsetLeft < -500) {
        console.log(node);
    }
}

And Firebug’s console spit out:

<ul id="cat-list">

Ah-ha! Now I was getting somewhere. A quick search through our CSS files for ‘#cat-list’ found an interesting line of code:

#categories.collapsed #cat-list {
    position: absolute;
    left: -999em;
    top: -999em;
}

And when JavaScript is turned on, the class ‘collapsed’ is added to the parent node #categories. In RTL mode this creates a huge scrollbar because -999em to the left of the page is a valid location that a user can scroll to. The solution?

.html-rtl #categories.collapsed #cat-list {
    position: absolute;
    left: 999em;
    top: -999em;
}

On any pages that are RTL, we add the class ‘html-rtl’ to the body tag in order to change the layout for RTL languages. This fixes the issue by moving the category list offscreen to the right, which is outside the page in RTL mode.

Things to remember:

• Firebug is your friend
• The DOM is a live document you can inspect, utilize this feature
• Be careful with positioning with sites that are LTR and RTL