Click here to get to the Mozilla Webdev “Ask Me Anything” on Reddit.

We will be there from 10am Central European Time to the evening, Pacific time, so in total about 18 hours. We hope this way people in most time zones will get the chance to get their questions answered.

Come by and ask us all you never dared to ask but always wanted to know!

Along with a few other Mozilla sites, we’ve been using pystatsd and Graphite for a while. This produces useful graphs of the site health and performance. For example, this graph plots HTTP responses we serve out from Python (this is not the amount of traffic to our site which is much larger).

To facilitate using the timing API we wrote a module between statsd and Django called django-statsd that, amongst other things, provides a way to interface between the browser and Python backend. The library has support for boomerang, a great front end timing library. I quickly wrote a much simpler service called stick, which sends a small subset of timings that we want.

Using stick is straightforward – once you’ve setup your Python backed and included your JavaScript, you call it in your page. For example:

stick.send('https://to.your.site/record')

The API returns with timestamps for each of the events. To give us meaningful numbers, we then subtract the timestamp for each of the events from navigationStart so we can tell the timings relative to that start point. And here’s the end result in our Graphite server:

For internal sites we collect 100% of the data. However, on our live site we get quite a lot of traffic, so we’ve rolled this out by collecting only 1% of traffic. We’ll start increasing this percentage once we are confident this won’t be causing problems. We’ll also be adding it to the about:addons page which gets quite a bit of traffic.

We’ll be tweaking things to ensure we pass through the right numbers and interpret them correctly. The amount of data is probably too low to be useful, right now, but I can’t help but wonder what the peak around 12.30pm today was…

As web developers at Mozilla, we take pride in making web applications that are used by millions of people every day. We work hard to promote the Open Web, both by showing off what the latest in web technology can do and by actively developing our projects in the open: as open-source projects on github.

Whenever we go to meetups, conferences, or chat with community members, people ask all sorts of questions, like:

• How do you make sure your websites are snappy for users across the globe?
• I found a bug in one of your websites and would like to help fix it! How do I get started?
• What frameworks do you use? And how many servers do you run certain projects on?
• Why are you all so good-looking?

To open the conversation to a larger community, we’ve decided to run an “IAmA” on Reddit, on Wednesday, January 18, 2012 Friday, January 20, 2012.

We’ll open a thread there in the morning, Central European Time, and will have plenty of people on the Webdev team participating until the evening, Pacific Time, so we should be able to touch most time zones one way or another.

Whether you are a seasoned Mozillian or a “newbie” in the community, whether you are a passionate Firefox advocate or pragmatic browser user with an inquiring mind, step by and talk to us! We’d love to hear from you.

We are Web Developers on the Mozilla Webdev team: Ask Us Anything.

For a link to the actual “AMA” post on reddit, visit this blog on January 18 January 20 or follow us on Twitter. If you have any questions, feel free to comment here or visit us in #webdev on irc.mozilla.org.

While over 85% of people using Firefox have at least one add-on installed, most people still don’t really know what an add-on is or the possibilities they provide. To fix this, the add-ons group teamed up with Mozilla Creative and an outside firm, Poke, to create a short video explaining what add-ons are and what they can do.

After seeing the video, we knew we had to do something awesome with Popcorn.js.

Watch The Video
On the page, click “Learn More” to play the video.

I recommend watching it in Firefox, since we interact with the browser in ways we couldn’t with Chrome. You can also view the video directly inside Firefox, by going to Tools > Add-ons > Get Add-ons > Click “Learn More” / “Learn More About Add-ons”.

Popcorn.js

For anyone unfamiliar, Mozilla’s Popcorn.js makes it easy to interact with video online using JavaScript. For anyone who has worked with jQuery or a similar library, it’s a breeze to pick up. Popcorn.js is the code part of Mozilla’s Popcorn, which is a larger project that aims to supplement video and audio with rich web content.

Setting up Popcorn.js is incredibly simple:

var pop = Popcorn('#promo-video');

Popcorn.js has a plugin called code, which lets you run arbitrary code when the video hits certain points.

pop.code({'start': start, 'end': end, 'onStart': function(){…}, 'onEnd': function(){…});

That’s basically all you need to know to get started with Popcorn.js. Everything else we did used regular JavaScript.

Like I mentioned, we only used one of the many plugins. Popcorn.js is incredibly powerful, and our video barely scratched the surface of what Popcorn.js plugins can do.

Showing Add-ons

We started by making add-ons fade in on the right side of the video when they are mentioned.

Note: You will only see this in the video if your browser window is wide enough.

Previewing Personas

We’re off to a good start. So far, we have add-ons showing up when they’re mentioned in the video. Now, wouldn’t it be cool if the video could interact with Firefox?

At one point the video talks about Personas, which are basically skins or backgrounds for Firefox. Since it would be incredibly annoying if any site you visited had the ability to skin your Firefox, the ability to trigger a preview is whitelisted to just the Mozilla add-ons sites.

Note: You can add other sites to this whitelist by going into Preferences > Security > Exceptions.

We thought it would be fun to style the browser with the Persona being mentioned — especially since we’re the only site that is allowed by the browser to do this. We simply fire an event when the video hits a certain point. To make sure the Personas are displayed instantly, we pre-load the Personas when you first click “Play”.

Blown Away

Later in the video, we see a plane fly across the screen. The papers and other items on the desk start to blow away as this happens. If you scroll down a bit, the add-ons below the video also start to flutter and are eventually caught in a JavaScript-y wind and blown away.

This is done using -moz-animation and -moz-transform: matrix(). Each add-on is given a class of “flutter” (starting at a staggered interval), and the add-ons flutter back and forth until they eventually “blow away”.

This isn’t Firefox-only; I just left out the Webkit code in the example for the sake of brevity.

In the next scene, both the backdrop in the video and the add-ons on the site drop back into place.

Other Things to Watch For

During the scene with the cameras, the page flashes as though a picture with a flash was being taken. The elements are simply hidden, then faded back in.

When the video talks about how add-ons can translate a page, some of the text on the page is translated along with the text in the video. (Note: This only works if you have fewer than three add-ons installed or you watch using this link. Otherwise, the top bar is not displayed.)

Uh-oh

We learned a few things along the way. We used popcorn-complete.js, which we didn’t realize loaded a few CSS, JS and SWF files off of popcornjs.org’s servers when the file was loaded. We mistakenly DOS’d popcornjs.org, which in turn made the add-ons site load slowly. Thanks to Jon Buckley and Rick Waldron, we were able to figure out what was going on and push a fix fairly quickly.

I also owe Chris and Krupa a big hug when I get back to the Mozilla office — they helped me out a ton by finding and fixing a bunch of bugs as we were pushing to production, since I was an unfortunate 30,000 feet in the air as the video was going live.

Looking Ahead

I have no doubt that Popcorn.js is going to become huge as HTML5 video takes off. News sites will use it to show relevant links and tweets, fashion sites will link to online stores as a celebrity is shown wearing a dress or shirt, video sites will provide closed captioning and annotations, media sites will use it to sync viewing across multiple computers — the possibilities are endless. I cannot wait to see what Popcorn powered videos people make next.

 Watch The Video
On the page, click “Learn More” to play the video.

Since the add-ons site is open source, be sure to view the code if you’re interested.

As for my family: not all is lost. The next morning I got an excited text from my cousin, who had watched the video. “Guess what I just did! I installed one of those picture addons on the top!” Sure, it pains me a bit that a Justin Bieber Persona now graces her Firefox — but hey, it’s a step in the right direction.

In Pootle, there are “Profiles” and that’s where the name and quick stats on contributions is stored. However, what I needed to do was to actually go through every “Suggestion”, “Submission” and “Review” and figure out who has contributed what on what project and language. Every such model has an overridden object manager that always joins on the “Users” model thus causing a ‘INNER JOIN’ on every simple data query. Thus this would build up an accurate page of who has contributed to what on the fly.

There are 14 projects, 99 languages, 847 users, 37,681 suggestions (and reviews) and lastly 127,226 submissions. The task is to group these first by language and then by project and lastly for each project, list the contributors names in alphabetical order. After some getting used to the data model and where things are located this wasn’t difficult. I built a prototype that visually represents this and showed it to the stakeholder who wanted this built.
The only problem is that due to the way the Django ORM and Pootle does this is that this takes on average: 11 seconds and involves 2,544 SQL queries. Yikes!

After some optimization (helped by my colleague Rob Hudson’s wonderful django-debug-toolbar) I was able to turn 11 seconds into 10 milliseconds (ie. one thousand times faster) and reduce the number of queries from 2,544 to 7! All it takes is just having to lessen some of the convenience of Django’s otherwise super practical ORM and get closer to the “SQL of things”. The code doesn’t express anything in raw SQL and everything is in Python with some very specific loops.

Why 7 queries and not 1, you might wonder. The reason is that the data I need to get to is spread across 7 different tables. Because some of them are large and joins are difficult what I do instead is pick only exactly what I want and need and nothing more. The logic kinda goes like this:

• Fetch all IDs and their Name (e.g. 52 and “Arabic”)
• For relevant tables with foreign keys, only fetch its ID, the foreign key ID and keep this in a list or dictionary whose structure implies information
• Sort and re-organize these lists and dictionaries in Python
• Summarize and replace the IDs with the Names for the final template rendering

I think the lesson here is that if your structures are big but not so big that you need a map/reduce job then getting closer to the raw SQL queries and doing the structure juggling in memory, you don’t need to replace your relational databases with NoSQL or typed raw SQL. Django’s ORM is fantastic. It’s slow because it chooses to be convenient but if it needs to be fast it’s just a few slight API calls away. If you’re curious, check out the code on Github.

]]> http://blog.mozilla.com/webdev/2011/12/15/django-optimization-story-thousand-times-faster/feed/ 2 http://blog.mozilla.com/webdev/2011/11/24/guest-post-i-want-to-contribute-how-do-i-start/ http://blog.mozilla.com/webdev/2011/11/24/guest-post-i-want-to-contribute-how-do-i-start/#comments Thu, 24 Nov 2011 19:56:04 +0000 Matthew Riley MacPherson http://blog.mozilla.com/webdev/?p=2287 Preface

This is guest post from one of webdev’s awesomesauce community contributors: Nigel Babu. nigelb contributes to a bunch of Mozilla’s web apps including Firefox Input and Socorro. He writes about Mozilla and open source on his blog and hangs out on #webdev on IRC — where we often talk about motorcycles.

---

At Mozcamp Asia, Tim Watts and I talked about contributing to Mozilla Webdev. When I met Tim, he asked me how I got started and what were some of the challenges I faced. This blog post is a summary of those challenges and a few solutions to help new contributors to Mozilla Webdev. This is also a condensed summary of our session, so if you missed it don’t feel too bad

Finding a Project

Finding a project to work on is the first baby step. Everything from here down is easier if you know what you want to contribute to; of course, it’s perfectly normal to be clueless as well. Everyone is mildly lost at this step at first. It helps if you have a clear understanding about what kind of code you want to write. Webdev has lots of different projects requiring different skill sets: Python/Django, JavaScript, HTML/CSS, and PHP are all in use at Mozilla. Almost all new projects require Python and Django knowledge, but we still have a few projects on PHP that you could help with — like mozilla.org and Socorro. Talk to us in #webdev with what kind of code you want to write, and we can help you find a nice project that needs help. You can start with a smaller project with fewer moving parts if you feel like it’s too much to take in. But one tip I have is: the bigger the project, the greater the opportunities.

“I Don’t Know What to Do”

It’s not easy to find something to do. I can attest to that. I know I had some trouble as well. Once you have found a project to help with, it becomes easier finding something to do. You could look at bugs with [good first bug] in the whiteboard — they’re generally good to start with. Another idea is to follow all the bugs for that project (bugmail can be noisy: set a filter or get ready for inbox explosion), so you can pick new bugs as they come in. Being in the project’s IRC channel also helps immensely. When fellow webdevs and Web QA learn that you are a new contributor and want something to do, they’ll be happy to point you to easy bugs or subscribe you to easy ones they see.

Finding a Mentor

While finding a mentor is not strictly necessary; however, it helps when you can ping someone to help you. When you find a project, some of the developers on that project are good candidates to mentor you. Feel free to ping the maintainers/developers for help when you are stuck. There are also the Stewards (https://wiki.mozilla.org/Stewards/Webdev) who can help you find a match: don’t feel shy about asking them to help you out.

Setting up Your Environment

Earlier, this was a challenging experience, sometimes it even seemed downright impossible. With the recent work we’ve done with vagrant, everything is much easier! Almost all new projects have a vagrant-based setup for the development environment and it should be much faster to get you off the ground. When in doubt, ask the project maintainers if there’s a vagrant setup for that project.

These were the 4 things that I faced and helped me start off. If you are interested in being a contributor and something ticked you off, talk to me — in the comments or on IRC. Feel free to reach us on #webdev on irc.mozilla.org with any questions or if you want to get started in contributing to Mozilla Webdev.

]]> http://blog.mozilla.com/webdev/2011/11/24/guest-post-i-want-to-contribute-how-do-i-start/feed/ 2 http://blog.mozilla.com/webdev/2011/11/21/git-using-topic-branches-and-interactive-rebasing-effectively/ http://blog.mozilla.com/webdev/2011/11/21/git-using-topic-branches-and-interactive-rebasing-effectively/#comments Mon, 21 Nov 2011 15:58:11 +0000 kumar303 http://blog.mozilla.com/webdev/?p=2245 When I first joined the webdev group at Mozilla I was a Mercurial refugee who had never used git or github. I was always daunted by git and suddenly I had to learn it really fast!  Fast forward to today and I can’t imagine working on a highly collaborative project without git or github.  Here is the workflow we use for the addons.mozilla.org project.  I highly recommend it and I’ll summarize exactly why at the end.  It’s pretty similar to how I’ve heard a lot of teams work but has some subtle differences.

Using topic branches

The first thing I do is sync up with master and create a topic branch for my new feature or bug fix:

git checkout master
git pull
git checkout -b add-email-to-install

Now I have a branch I can commit code into without affecting master.  Git checkout makes it super easy to switch between branches in the same repository clone if I’m multi-tasking or applying hot fixes.  In addition to git checkout, you can also use git stash to switch tasks.

Commit messages

It’s really important to write a well-formed git commit message. We always include a ticket number into bugzilla, our tracker, so that anyone can get the full back story about a change.

Ask for a code review

Once I’ve added my feature with passing tests I commit my changes, push to my personal fork of the repository, and ask someone on my team to review the code.  On addons.mozilla.org we just ping each other in IRC with a link to the commit or a link to the compare view.  If no one is around we submit a pull request.

Github has a sweet interface where you can write comments directly on the diff, like this:



Whoops, another change is needed based on feedback from the code review.

Fixing up the topic branch

The nice thing about working in a topic branch is it’s isolated from master and no one else is tracking that branch so I can use git rebase to create the best commit before merging into master.  Let’s say I have some commits on my branch like this:

$ git log --pretty=oneline -2
825d662cc69774e412119e1eb7ae0900c29d89a0 Fix: put code in a transaction
31378788f321b46f5e27f9fb51bdd19365636871 Adds email to the install record (bug #NNNNNN)

What I really want is to combine those two commits into one.  I can do that with git rebase —interactive. I type:

git rebase -i HEAD~2

Then I’ll get a prompt for rebasing my last two changes:

pick 3137878 Adds email to the install record (bug #NNNNNN)
pick 825d662 Fix: put code in a transaction

# Rebase 194b59d..825d662 onto 194b59d
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

If I put the word fixup next to my second commit, it folds it into the first:

pick 3137878 Adds email to the install record (bug #NNNNNN)
fixup 825d662 Fix: put code in a transaction

Now I have one commit (it’s actually a new commit) that contains all of my changes:

$ git log --pretty=oneline -1
c7846808c8296dd49d49612101aaed7cdfd6d220 Adds email to the install record (bug #NNNNNN)

Pretty slick, right?

Typically you’d want to wait until everyone has had a chance to review your code before you start rebasing.  However, git pull requests do handle rebased changes.  You can push -f to your own fork and the pull request will remove the old commits from the conversation and add the new ones at the bottom.

Merge into master

When my changes are ready, I can merge my branch back into master.  However, I don’t need to make a merge commit if there’s only one commit to merge in.  That would clutter up the logs.  I can do this with a fast-forward merge:

git checkout master
git merge --ff add-email-to-install

Now I can close the ticket in our tracker with a direct link to my changes.

Sometimes I might actually make multiple commits on a single topic branch.  In this case I would want to retain the automatic merge commit.  That is, I wouldn’t do a fast forward merge in the case of multiple commits:

git merge --no-ff add-email-to-install

I can then close the ticket with a link to the single merge commit that shows all changes introduced by the branch.

Fixups, for ninjas

If you follow this pattern you’ll become accustomed to frequently fixing up your topic branch. I created a ninja alias for it in ~/.gitconfig like this:

[alias]
    ...
    fix = "!f() { git commit -a -m \"fixup! $(git log -1 --pretty=format:%s)\" && git rebase -i --autosquash HEAD~4; }; f"

When on a topic branch with uncommitted changes I can then type:

git fix

That will automatically commit my change and pre-configure the rebase prompt to fold it into the last commit.

UPDATE: AMO team member Allen Short pointed out in the comments that git fix is the same as git commit -a --amend if you’re just fixing up the last commit. Awesome!

Synchronization with master, for ninjas

If you’re on a project that has a lot of commit activity you’ll probably want to rebase your feature branch on top of master often.  I added a ninja alias to ~/.gitconfig for that too:

[alias]
    ...
    sync = "!f() { echo Syncing $1 with master && git checkout master && git pull && git checkout $1 && git rebase master; }; f"

When I’m on my feature branch and I want to synchronize it with all the latest changes on master, I type:

git sync add-email-to-install

The main benefit to syncing a branch before merging into master is that a fast-forward merge won’t create a new commit.  This helps you safely delete work branches later on since it won’t look like you have un-merged changes.  It’s also useful to do a last minute spot check before merging into master: do the tests still pass? do I need to adjust my SQL migration script? etc.

UPDATE: Fernando Takai posted a simpler version of this in the comments using git checkout - to go back to the last branch you were on. You can then simply type git sync from the branch. Thanks!

Why resort to all these ninja like git strategies?

• Using git blame on a single line of code is more likely to give your team a full picture of all the reasons why that line of code was introduced. For this same reason, we at addons.mozilla.org always link to our bug tracker in each commit.
• Your commit log will have a high signal to noise ratio making it easier to skim when looking at a compare view between releases.
• Ninjas don’t make mistakes.  Ever.

Random Notes

• Kernel hackers frown on using rebase but that’s probably because many people are committing to the same files and it’s important to see what the original starting tree was when work on a new feature started.  For web development, if two members on your team are working on the same line in the same file then your team isn’t communicating well enough.  I rarely see conflicts on my team that aren’t resolved automatically by a three way merge.
• After committing to master you might discover a mistake.  That’s fine, make a new commit.  Be sure to never fixup a commit on master because everyone tracking master will be sad!
• Where do your fixed up commits go?  They are still there but are detached from any branch and thus get deleted eventually by git’s garbage collector.
]]> http://blog.mozilla.com/webdev/2011/11/21/git-using-topic-branches-and-interactive-rebasing-effectively/feed/ 12 http://blog.mozilla.com/webdev/2011/11/18/scrubbing-your-django-database/ http://blog.mozilla.com/webdev/2011/11/18/scrubbing-your-django-database/#comments Fri, 18 Nov 2011 23:05:38 +0000 Andy McKay http://blog.mozilla.com/webdev/?p=2206 This is the second in a series of posts, focusing on issues around open sourcing your Django site and data privacy in Django.

You’ll end up with production data in your Django database and that will likely contain different kinds of data such as: configuration data, required basic data (categories for example), collected data and personal user data. There’s a couple of reasons for taking that production data and copying it off your production servers:

• for developers and contributors you want a sample copy of the app with some key data in.
• for testing or staging servers, you might want to copy down the database from the production server so you can test certain scenarios or load.

Extracting parts of your database

For the first case, it’s nice to prepare a minimal copy of the database that contains key data. For example, for those wanting to develop or contribute to addons.mozilla.org we have Landfill by Wil Clouser.

Django comes with a nice facility for loading data, fixture dumping and loading. This can be used to pull data out of your database and then reload it. However the built in Django dumpdata dumps all the records for your model (depending upon your default object manager). That might not be what you want for this scenario. So a useful utility for dumping just the records you want is provided Django fixture magic written by Dave Dash.

A standard dumpdata looks like this:

manage.py dumpdata users.UserProfile

And will dump every UserProfile. By contrast:

manage.py dump_object users.UserProfile 1

Will just dump the UserProfile with primary key of 1. Django fixture magic also has a few other useful things such as merging and reordering fixtures.

This allows you to trim a set of fixtures from your live database down quickly. Then developers or contributors can load the key parts of the database that they need from those fixtures.

Anonymising the database

Sending the production database downstream to internal developers or internal test sites is a pretty common use case. This process does not require a complete clean of the database, but it does require some cleaning of database. If you stored credit card data, for example, you’d never want to copy that off your production database.

At Mozilla we use an anonymising script, written by Dave Dash again. There are few options: to truncate, nullify, randomize or selectively delete. The format is a simple YAML file, for example:

   tables:
        users:
            random_email: email
            nullify:
                - firstname

This is a snippet from the config script for addons.mozilla.org.

When the IT copies the databases down from production, this script is run against the database. Ensuring that when us developers access the backups to investigate certain issues, we’ll be getting the bits we want and not the bits that might expose user data.

In the next blog post we’ll look at logs and tracebacks.

]]> http://blog.mozilla.com/webdev/2011/11/18/scrubbing-your-django-database/feed/ 1 http://blog.mozilla.com/webdev/2011/11/10/localtunnel-on-actual-mobile-devices/ http://blog.mozilla.com/webdev/2011/11/10/localtunnel-on-actual-mobile-devices/#comments Thu, 10 Nov 2011 17:36:25 +0000 Peter Bengtsson http://blog.mozilla.com/webdev/?p=2209 Surely you’re convinced too now that developing for the mobile web is important. It certainly is for us here at Mozilla. Mobile web development isn’t just about making a ultra-slim stripped down version with no images or Javascript widgets. It’s also about taking regular full powered web applications and making stepwise adjustments to make sure they work responsively on all mobile devices which means everything from tablets to regular cell phones over 3G.

There are numerous good guides such as this for Designing for Mobile Devices and there are great add-ons for Firefox such as the User Agent Switcher. And don’t miss Jason’s wonderful blogs from the archive part 1, part 2, part 3 and part 4. But ultimately, nothing beats actually testing it on a real physical device.

That’s where localtunnel (by Jeff Lindsay) comes in. What it does is that it binds a local port (e.g. port 8000) on your desktop to a real domain name (e.g. xyz123.localtunnel.com) all over SSH. So, you start your local Django server locally and then start a local tunnel and after waiting 1-2 seconds you get a real domain name to type into your iPhone, Android, Nokia smartphone, whatever and you can test away as much as you like.

Granted, you can use it not just for mobile device development but for anything such as telling a colleague over IRC to visit your site running locally on your laptop. And also, if you are in an office where you connect to the same local WiFi with your laptop as you do with your mobile device then you can just use a local IP address. But that’s often not the case especially when working with people in different locations.

An added, yet strange, benefit with using localtunnel to test your mobile device is that it is naturally slow. That’s because the traffic has to go via SSH over the network and your local network won’t be as fast as a real data center. However, in real life, the network connection will be slow when using a mobile device so it’s important to see how your mobile web app behaves with a limited bandwidth.

]]> http://blog.mozilla.com/webdev/2011/11/10/localtunnel-on-actual-mobile-devices/feed/ 2 http://blog.mozilla.com/webdev/2011/10/25/go-speed-racer/ http://blog.mozilla.com/webdev/2011/10/25/go-speed-racer/#comments Tue, 25 Oct 2011 23:15:29 +0000 Gregory Koberger http://blog.mozilla.com/webdev/?p=1994 Before pushing our CSS and JavaScript assets to our CDN, we run them through jingo-minify to concat and minify the files, as well as cache bust them and any resources (such as images) contained inside them.  Turns out, this was by far the slowest part of our push process — it took  between 80 and 160 seconds for addons.mozilla.org (AMO) assets.  It wasn’t a huge priority, since most of the time this happens in the background and nobody really notices.  However, I wanted to see how fast I could get it.

Note: All times are based on AMO being compressed on a MacBook.  On our build server, those 80 seconds took about 160 seconds.  Most times are averages and/or estimates, and are based on an initial 80 seconds.


UglifyJS and clean-css (-43s)

This was the low hanging fruit.  YUICompressor is slow.  We managed to cut off over half our original time by switching to UglifyJS for JavaScript and clean-css for CSS, both of which are written in Node.  This switch means less errors (we were getting a lot with YUI), smaller files and much faster build times — and none of us are sad to see our Java dependency disappear.  (I benchmarked all our options before we chose them.  If you’re interested, you can run them yourself.)

Unlike YUICompressor, UglifyJS builds an abstract syntax tree, compresses the AST, then converts the whole thing back to (minified) JavaScript.  So far, this has meant that UglifyJS has been much more resilient against our sometimes-misformed JavaScript. Missed semicolons no longer take down the whole site’s JavaScript. The build times are about the same by my benchmarks, but the results are far superior. [JS Benchmark results]

Clean-css works much like YUI (lots of regex), which is a bit unfortunate.  However, it does so much faster — 10x as fast, to be exact.  [CSS Benchmark results]

Original Time: 80s
New Time: 37s
Difference: -54%

Saving File Hashes (-2s)

We were fetching the hash (originally the git commit id, now hash of the actual file) for the same files fairly often.  By sticking the already-computed hashes into a dict, we were able to shave off a small but significant amount of time.

Original Time: 37s
New Time: 35s
Difference: -11%

Use File Hashes For Image Cache Busting (-12s)

When I first wrote cache busting for individual images in our CSS files, I used the image’s most recent git commit as the cache-busting hash.  As it turns out, your script gets really slow when you’re looking up the history for dozens of images using GitPython.  So, I switched to using a hash of the file.  The end result is the same — the hash is only updated if the image is changed.  However, using file hashes instead of git commits shaved a good chunk of time off our build process.

Original Time: 35s
New Time: 23s
Difference: -34%

Only Minify When Necessary (up to -17s)

I got down to 23 seconds, and was stuck.  The biggest time sink in the process was the minifying of files, which I couldn’t do much about without touching the UglifyJS or clean-css code.  Even if I wanted to, I doubted I could do much to speed either of them up.  After thinking about it for a bit, I realized that there was no reason for us to be minifying every file each time.  We only needed to minify the files that had actually changed.  So, jingo-minify now writes the concatted file out to a temporary location, and compares it to the preexisting concatted file.  If they’re the same file, we have no need to re-minify.

Since we push to preview every time anyone commits and most commits don’t modify more than one or two JS or CSS files (if any), we can save between 15 and 17 seconds each time.

Original Time: 23s
New Time: 6s
Difference: -74%

Overall Results

So, that’s how I managed to speed up our asset compression by over an order of magnitude — from 80 seconds down to about 6.  (Note: Time is based on the script on a MacBook.  On the slower build server, it’s closer to 160 seconds down to 9.)  Hopefully this means much smoother pushes for AMO, SUMO and anyone else using jingo-minify to compress their assets.

]]> http://blog.mozilla.com/webdev/2011/10/25/go-speed-racer/feed/ 3