Using topic branches

The first thing I do is sync up with master and create a topic branch for my new feature or bug fix:

git checkout master
git pull
git checkout -b add-email-to-install

Now I have a branch I can commit code into without affecting master.  Git checkout makes it super easy to switch between branches in the same repository clone if I’m multi-tasking or applying hot fixes.  In addition to git checkout, you can also use git stash to switch tasks.

Commit messages

It’s really important to write a well-formed git commit message. We always include a ticket number into bugzilla, our tracker, so that anyone can get the full back story about a change.

Ask for a code review

Once I’ve added my feature with passing tests I commit my changes, push to my personal fork of the repository, and ask someone on my team to review the code.  On addons.mozilla.org we just ping each other in IRC with a link to the commit or a link to the compare view.  If no one is around we submit a pull request.

Github has a sweet interface where you can write comments directly on the diff, like this:

Whoops, another change is needed based on feedback from the code review.

Fixing up the topic branch

The nice thing about working in a topic branch is it’s isolated from master and no one else is tracking that branch so I can use git rebase to create the best commit before merging into master.  Let’s say I have some commits on my branch like this:

$ git log --pretty=oneline -2
825d662cc69774e412119e1eb7ae0900c29d89a0 Fix: put code in a transaction
31378788f321b46f5e27f9fb51bdd19365636871 Adds email to the install record (bug #NNNNNN)

What I really want is to combine those two commits into one.  I can do that with git rebase —interactive. I type:

git rebase -i HEAD~2

Then I’ll get a prompt for rebasing my last two changes:

pick 3137878 Adds email to the install record (bug #NNNNNN)
pick 825d662 Fix: put code in a transaction

# Rebase 194b59d..825d662 onto 194b59d
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

If I put the word fixup next to my second commit, it folds it into the first:

pick 3137878 Adds email to the install record (bug #NNNNNN)
fixup 825d662 Fix: put code in a transaction

Now I have one commit (it’s actually a new commit) that contains all of my changes:

$ git log --pretty=oneline -1
c7846808c8296dd49d49612101aaed7cdfd6d220 Adds email to the install record (bug #NNNNNN)

Pretty slick, right?

Typically you’d want to wait until everyone has had a chance to review your code before you start rebasing.  However, git pull requests do handle rebased changes.  You can push -f to your own fork and the pull request will remove the old commits from the conversation and add the new ones at the bottom.

Merge into master

When my changes are ready, I can merge my branch back into master.  However, I don’t need to make a merge commit if there’s only one commit to merge in.  That would clutter up the logs.  I can do this with a fast-forward merge:

git checkout master
git merge --ff add-email-to-install

Now I can close the ticket in our tracker with a direct link to my changes.

Sometimes I might actually make multiple commits on a single topic branch.  In this case I would want to retain the automatic merge commit.  That is, I wouldn’t do a fast forward merge in the case of multiple commits:

git merge --no-ff add-email-to-install

I can then close the ticket with a link to the single merge commit that shows all changes introduced by the branch.

Fixups, for ninjas

If you follow this pattern you’ll become accustomed to frequently fixing up your topic branch. I created a ninja alias for it in ~/.gitconfig like this:

[alias]
    ...
    fix = "!f() { git commit -a -m \"fixup! $(git log -1 --pretty=format:%s)\" && git rebase -i --autosquash HEAD~4; }; f"

When on a topic branch with uncommitted changes I can then type:

git fix

That will automatically commit my change and pre-configure the rebase prompt to fold it into the last commit.

UPDATE: AMO team member Allen Short pointed out in the comments that git fix is the same as git commit -a --amend if you’re just fixing up the last commit. Awesome!

Synchronization with master, for ninjas

If you’re on a project that has a lot of commit activity you’ll probably want to rebase your feature branch on top of master often.  I added a ninja alias to ~/.gitconfig for that too:

[alias]
    ...
    sync = "!f() { echo Syncing $1 with master && git checkout master && git pull && git checkout $1 && git rebase master; }; f"

When I’m on my feature branch and I want to synchronize it with all the latest changes on master, I type:

git sync add-email-to-install

The main benefit to syncing a branch before merging into master is that a fast-forward merge won’t create a new commit.  This helps you safely delete work branches later on since it won’t look like you have un-merged changes.  It’s also useful to do a last minute spot check before merging into master: do the tests still pass? do I need to adjust my SQL migration script? etc.

UPDATE: Fernando Takai posted a simpler version of this in the comments using git checkout - to go back to the last branch you were on. You can then simply type git sync from the branch. Thanks!

Why resort to all these ninja like git strategies?

• Using git blame on a single line of code is more likely to give your team a full picture of all the reasons why that line of code was introduced. For this same reason, we at addons.mozilla.org always link to our bug tracker in each commit.
• Your commit log will have a high signal to noise ratio making it easier to skim when looking at a compare view between releases.
• Ninjas don’t make mistakes.  Ever.

Random Notes

• Kernel hackers frown on using rebase but that’s probably because many people are committing to the same files and it’s important to see what the original starting tree was when work on a new feature started.  For web development, if two members on your team are working on the same line in the same file then your team isn’t communicating well enough.  I rarely see conflicts on my team that aren’t resolved automatically by a three way merge.
• After committing to master you might discover a mistake.  That’s fine, make a new commit.  Be sure to never fixup a commit on master because everyone tracking master will be sad!
• Where do your fixed up commits go?  They are still there but are detached from any branch and thus get deleted eventually by git’s garbage collector.

Shortly after I started at Mozilla in April, I was assigned my first project — developing the replacement for AMO’s statistics pages for add-on developers. The current page was in need of both a visual and functional refresh. One of the great things about working on AMO is that we can make certain expectations as to what browser our registered users and developers are running — namely, a recent version of a modern browser. This was exciting because it meant the new stats pages could use next-generation web features to deliver a faster and more feature-rich experience to our users. I’m excited to say that, though they’re not ready to replace the old pages quite yet, the new stats pages are fit for public preview. Take a look here!

One of the biggest challenges on the stats pages is the management of data. We store our add-on performance statistics in what is essentially a CSV format: a set of values for each metric and date. Since performance data is historical data, we would wind serving overlapping sets of data for each pageview. I realized that if a user visiting the page today comes back tomorrow, there’s only one day of new data they haven’t yet seen. The solution? localStorage!

When a user visits a page for the first time, they download the data for the default time frame (which, right now, is the past 30 days). The next time they come back, the data is fetched from localStorage instead, and the page renders much, much faster. How does this work? When the user requests a certain time range of data, the page looks at the range of time for which it already has data. If the request is satisfied by the local data, we can skip an AJAX call. Subsequently, if any of the time range requested is not available locally, the server is asked only for the portion that is missing. This allows for the user to be able to use the stats pages frequently with very little additional server load.

Having a fast persistent layer for raw data is all well and good, but analysis and visualization are what make the stats pages useful. For that I turned to another fantastic new web technology — <canvas>. We settled on using the exceptionally nifty Highcharts charting library, which utilizes <canvas> and inline SVG (with graceful fallbacks) for its rendering.

The charts, as well as all the other data-driven UI, each have their own pageview-length cache layers to store the generated objects they use internally, such as chart configuration objects and individual pages of data. If a particular object hasn’t already been generated, we request the data (going to the server if necessary) and then store the fully generated object for the length of the pageview. This allows for fast switching of data views within the page with minimal lag. The caches use a client-side cache object. that generates and accepts callbacks, to allow asynchronous actions such as AJAX requests to occur. I’ve posted the code for the object here for those interested.

While the charts use the time-series data from the server fairly directly, a number of UI elements are presenting aggregate statistics on the raw data in the form of sums averages and ranked lists. Instead of making another request for data the page essentially has, I decided to compute aggregations locally. Realizing iterative computation is a good way to send the browser on a one-way trip to Beachball Town, I found an excellent opportunity to use Web Workers to perform those tasks in the background. The raw data is sliced, time-wise, into the needed chunk, which is passed off to a Web Worker thread. The result? The computationally expensive job of iteratively manipulating the data is off the main thread, where it refrains from putting a chokehold on the browser, and has the added benefit of running much faster in a dedicated thread.

In the initial implementation, I found that, at times, more than 10 workers were being spawned at a time to perform background tasks. I can count on one hand how many people I know who own a computer with more than 10 CPU cores, and I am not in that group. It wasn’t really any more efficient, and the CPU usage was hairy to boot. To address this, I wrote a simple Web Worker pool that queues jobs and distributes them to a limited number of workers. Much better.

Lastly, one of the goals for the new stats pages is the ability to deep-link or bookmark a particular view of the stats. On browsers that support it, the URL for the current view is being updated using the new history.pushState(). Of course, we fall back to location.hash when the fancier API isn’t available.

The process of building the new stats pages led me to this realization: A lot of the demos of HTML5 and its associated trappings spend great effort trying to visually dazzle, and it’s way too easy to write it all off as nothing but eye candy. What people forget is that a lot of the new features in modern web browsers have practical ways to make the development process cleaner, the user experience faster, and the end-result more sophisticated. Though truth be told, prettier pages aren’t so bad either.

^{* and CSS3, and next-generation JS APIs.}

Notable changes are:

• Migrating from CakePHP to Django
• Moving from SVN to Git
• Continuous integration
• Faster deployment
• Processing data offline
• Improved documentation

Take a look, it’s a good overview of the technical challenges of managing a large and complex website at an enormous scale.

To summarize the issue, for no apparent reason in right-to-left languages a really long scrollbar would appear at the bottom of the window.

Even though there was a scrollbar, when you scrolled all the way to the left, nothing was there. Another reason this was odd was the scrollbar only appeared in right-to-left (RTL) languages. Inspecting the page via Firebug didn’t give any clues as to what was causing the issue as there was no element hidden somewhere onscreen. Finally, to make things even weirder, the scrollbar only appeared when JavaScript was turned on.

After some thought, I had a feeling that we might be using absolute positioning to position an element to the left and above the page offscreen, which is quite common. In a RTL page, however, left is does not move an element outside a page’s boundaries. So the result is you get a scrollbar.

So what’s a web developer to do? Firebug to the rescue! I popped it open and started typing some JavaScript into the console to find an element that seemed really far offscreen:

var nodes = document.getElementsByTagName("*");

for(var i=0; i < nodes.length;i++) {
    var node = nodes[i];
    if(node.offsetLeft < -500) {
        console.log(node);
    }
}

And Firebug’s console spit out:

<ul id="cat-list">

Ah-ha! Now I was getting somewhere. A quick search through our CSS files for ‘#cat-list’ found an interesting line of code:

#categories.collapsed #cat-list {
    position: absolute;
    left: -999em;
    top: -999em;
}

And when JavaScript is turned on, the class ‘collapsed’ is added to the parent node #categories. In RTL mode this creates a huge scrollbar because -999em to the left of the page is a valid location that a user can scroll to. The solution?

.html-rtl #categories.collapsed #cat-list {
    position: absolute;
    left: 999em;
    top: -999em;
}

On any pages that are RTL, we add the class ‘html-rtl’ to the body tag in order to change the layout for RTL languages. This fixes the issue by moving the category list offscreen to the right, which is outside the page in RTL mode.

Things to remember:

• Firebug is your friend
• The DOM is a live document you can inspect, utilize this feature
• Be careful with positioning with sites that are LTR and RTL

Thanks for your help!

Please let us know if you see any regressions or anything you’d like to see changed.

The next update is scheduled for May 29th and will be mainly a bug fix release with minor new features (like bug 432669). There are currently 28 bugs scheduled for the 3.4.3 push.

We’ll be committing the last of the patches very shortly and the changes will be available on preview.addons.mozilla.org. Feel free to look at the changes and send us any feedback you have. We expect to push the changes live this Thursday evening (May 1st).

Our next release, 3.4.2, is currently scheduled for May 15th. 3.4.2 will also be a bug-fix release – there are currently 19 candidates.

Our next release will be AMO 3.4.1, the first of three dot releases for AMO 3.4, which is our next milestone to be completed before Firefox 3.