Taras’ Blog

Linux Plumbers

A few of us attend LinuxPlumbers last month, I gave talk. Basic summary: kernel + toolchain could add a few straight-forward features to make Firefox and other apps start faster and use less memory.

Performance Team

I am no longer alone on my quest of performance whack-a-mole. Since spring I’ve been growing a team of extremely talented people to help make Firefox faster and slimmer. As a manager I do more people tasks now, so I have less ” ____ sucks” blog posts to write. We are currently working on massively speeding up Fennec startup via incremental decompression (sort of like an efficient UPX for android), fixing SQLite + other IO usage in Firefox, reordering binaries for faster startup, faster shutdown. The idea is to focus on all areas of browser performance other than rendering the web (for now).

Performance Help

“Perf work is cool, but I wouldn’t know where to start with finding a good first perf-bug” <– is this you? Would any of my readers be interested in a roundup of next set of “low hanging fruit” perf-improvements to be had?

Example bugs:

• Implement readahead support on Android
• Use readahead for omnijars

Firefox 7 marks a turning point in how we measure Firefox performance. Traditionally we measured Firefox performance on individual developer machines and our build & release infrastructure. However it turns out synthetic benchmarks do not correspond to real-world Firefox usage: it is difficult to model a “typical” computer in a lab environment. Surprisingly slow consumer hardware, changes in usage patterns, preinstalled bloatware all affect Firefox performance in surprising ways.

Firefox 7 telemetry will prompt users to opt-in to reporting performance data to Mozilla. This data will supplement our existing benchmarking infrastructure to help us optimize future Firefox releases. Telemetry performance metrics are very lightweight and will not negatively impact Firefox performance.

In addition to transmitting data via SSL, Mozilla privacy team worked tirelessly to ensure that no personally-identifiable information is sent via telemetry. Whereas many other software projects stamp this kind of data with a unique per-user id, we opted for a per-session id which is reset every time the browser restarts. Telemetry is also disabled while in private-browsing mode.

The following telemetry data will be gathered in Firefox 7:

• Memory usage
• CPU core count
• Cycle collection times
• Startup speed

Use the about:telemetry extension to check on your browser performance. The following screenshot shows how to enable telemetry:

I’m very excited that Firefox finally joins the ranks of cars, airplanes and other software projects in making performance decisions based on real evidence gathered in the wild.

Firefox 7 features improved startup speed. Our research has shown that most OSes are not capable of starting large programs efficiently (see my older blog entries). As a result, Firefox 7 will explicitly tell the OS to aggressively preload our xul.dll/libxul.so/XUL library before passing it on to the runtime linker. This marks the productization of the approach explored in “20-line patch that doubles Firefox startup” that got people so excited. See bug 552864 and dependent bugs for exciting technical details.

Do Not Try this at Home: Ugly Windows

Note: Windows Prefetch does everything possible to thwart exciting startup optimizations. Above optimization only works when prefetch is disabled/broken (ie Firefox should be faster with Windows Prefetch off). See this comment on disabling prefetch for Firefox. Alternatively one may get Windows Prefetch to not slow down Firefox startup by the following magic incantation:

1. Install Firefox (or delete the windows prefetch entry for existing Firefox 7)
2. Reboot (Do not start Firefox after installation!)
3. Start Firefox.

Above helps populate the Windows Prefetch in a less counter-productive way. Explanation: on warm startup Windows Prefetch records irrelevant IO operations and blocks Firefox startup to preload files that Firefox accesses after startup.

Note: the heuristic that we use to detect Windows Prefetch will also prevent this optimization from kicking in on some exceptionally slow hard drives when prefetch is off. This is unfortunate as this optimization is most dramatic on exceptionally slow machines.

According to our preliminary telemetry data, less than 25% of our Windows users have Windows Prefetch off and can benefit from vastly improved startup. We expect to improve this in future Firefox releases by scheduling a prefetch cleanup operation similar to the script in above bugzilla comment.

Operating systems without counter-productive startup heuristics (ie Mac/Linux) will simply allow Firefox 7 to start faster.

I like having guests who bike. So when Brian came over I decided this would be a great opportunity to explore going to the coast and back by bike (velodirt style). The following is a map of our trip (190miles, 2 nights).

The gravel roads were somewhat challenging. It was hard to climb in the heat so at times we had to push our bikes.

Things got more unpleasant when we encountered loose gravel by Tillamook which sent Brian sliding over the edge

Amazingly he got off without a single scratch (type-inference will go on!).

Eventually we got to Cape Lookout.

The next day we biked to and spent the night in Nehalem Bay.

Then it was time to head back

On the way back we did some more gravel roads, this time they were really nice. The road took us through a berry-laden forest and rewarded us with some pretty awesome views.

(If you squint you can see Brian)

The thing to keep in mind about gravel roads is that they like to get out of sync with the maps. We had to reroute due to roads going in a different direction, and due to various kinds of deadends.

In one extreme case a road was clearly on the map, but had a “DEAD END” sign on it. Which got us thinking “is this a dead end for cars or is it for real?”. After some climbing, the gravel road reduced to a nice forest path.

Further on it became obvious that someone tried hard to kill this road. Some road hater put up over a dozen tree pile barriers, with some reaching my height.

Turned out the road did indeed go through. Perhaps some day someone will start a cyclocross race dedicated to bringing this road back

The last highlight of the trip was this delicious-looking buffalo herd.

I love cycling in Oregon.

Static analysis can be a very fun pastime. One gets to sift through giant codebases looking for interesting clues, what could be more fun? A couple things qualify: a) static analysis accompanied by cool rewrites b) static analysis accompanied by cool visualizations.

Cool Rewrite

Michael Wu’s boolcheck tool is awesome. He wrote it to check that “typedef int” bools are really being used as booleans and aren’t perversely carrying integer values. The process of writing the tool is cool. As Michael is discovers bugs/disagreements stemming from setting “typedef bool PRBool”, he just adds another pattern to check for to the tool and never has to worry about that pattern again. I hope to see someone apply boolcheck to the linux kernel, GTK projects or anything else with int booleans. Some projects don’t have the luxury of switching to real bools, so they can continue using a static checker to make up for it.

Pretty Code

I’ve blogged about DXR many times. As of this week clang-based DXR is on par with the old Dehydra-based one. Callgraph, inheritance, etc queries now work. Joshua did an outstanding job gutting and rewriting the DXR backend this summer and is now going back to school. I’m extremely impressed with his work this summer. I didn’t think it was possible to get as far as he did.

We are looking for more help with DXR. Please deploy it on your pet project, contribute plugins for various languages, simplify deployment, etc.

Additionally, now that the backed is in a fairly decent shape, we are looking for someone to help us turn DXR into the slickest code browsing tool ever(we have some ideas written down). I’d like interactive graphs, various code visualizations, integration with bugzilla, etc. This needs a JSON-query frontend and a few other bits & pieces to be implemented.

Interns Wanted

We would love to hire more static analysis interns. Are you student who dreams about making large codebases easy to grasp? Do you want to spend a few months making Control Flow Graphs behave? If that sounds like your calling: leave a comment, send me an email.

Telemetry Present

Telemetry infrastructure has only been deployed for week, but we are already gathering interesting data:

• Memory usage from about:memory gives us an idea of what constitutes typical memory usage for Firefox
• Cycle-collection overhead, stats tell us about browser pauses due to memory cleanup
• Detailed startup profiling tells us whether our new library preloading logic is effective
• Info on whether the browser was shutdown correctly will help us diagnose shutdown problems
• Plugin enumeration timing to make sure my faster plugin enumeration stays fast
• HTTP connection profiling to help optimize page loads

Since the branch point for Aurora is approaching in less than a week, I don’t expect many more probes for Firefox 7.

Telemetry Future

One problem with doing awesome optimizations that as code changes, they frequently get accidentally undone. I plan to add telemetry to keep tabs on every significant optimization that I do in the future, in addition to retroactively adding it to the past ones. I expect other Mozilla developers to do the same.
In addition to keeping tabs on Firefox performance, we can also learn about JavaScript feature adoption, the kind of hardware (and OS) that users run Firefox on, etc to better match users’ needs.

I was pleased to see Mozilla static analysis mentioned on lwn. Yes indeed, the mailing list has been pretty dead (most of our communication happens on irc.mozilla.org #static). I completely failed to build a community around my static analysis tools. Perhaps more people will try Dehydra now that it’s getting into Debian. The hydras are still alive, evidence can be seen in the mercurial commit log. Development has slowed because the hydras are now considered to be feature-complete and my primary focus is elsewhere in Mozilla now.

As to why open source static analysis has failed to take off, I have a few theories. I think the main problem is that static analysis requires a compiler/correctness/type-system-nerd/large-scale-development-nerd type personality. That’s a pretty rare intersection of hobbies to begin with. One also has to hate the stone age that C/C++ ecosystem we are in, but not move on to shiny new Haskell/Ocaml/whatever communities.

Have I failed at igniting the static analysis revolution?

1. My goal primary goal was: provide a way to analyze Mozilla source code to speed up our development + refactoring efforts.
2. My secondary goal was to make sure that whatever work I do, nobody else has to suffer through the unbelievably sucky infrastructure cruft I had to work through.
3. Lastly, I did put in some effort at promoting open source static analysis (by giving talks at conferences, etc) since working in an active community is more fun.

Mozilla side:

I’m happy to report that I achieved a culture shift at Mozilla. Instead of people saying “oh god, I can’t find all instances of ___ issue in 3million lines of C++ code”, it’s pretty common to hear “lets solve this through static analysis”. Dehydra was designed to take the bitchwork (boilerplate of compiler integration, etc) out of static analysis so one can focus on the analysis part. New Dehydra users within Mozilla seem to confirm that.
Instead of pondering whether certain tool-assisted refactorings are feasible, we plan to embark on some now (turned out we were understaffed to keep up with tool output and overburdened by api compatibility before; more on this in a future blog post).

No More Static Analysis Bitchwork:

The worst aspect of dealing with C++ is parsing it. The second worst aspect is dealing with the preprocessor. With respect to parsing C++ we went from weirdo-custom-frontends(ie Elsa, EDG, etc) and “GCC will never allow plugins, don’t waste your time” to GCC adopting a plugin architecture that suited my static analysis needs. I also implemented source-location transformation tracking(-K) in mcpp, so nobody has to suffer through undoing braindamage inflicted by the C proprocessor again.
I hear at least a couple of people benefited from MCPP work and I take partial credit for every new analysis GCC plugin. I suspect I saved a few person-months for somebody

Btw, I think Chris Lattner’s from-scratch effort on Clang is way awesomer than anything I could ever accomplish.

Conferences & Stuff:

I admit complete and utter failure in this regard. Most open source people have low regard for static analysis. Linus seems to take a million-monkeys-with-type-writers approach (ala the open source eyeballs approach to security) to ensuring kernel code quality (which is a reasonable approach when you have mobs of contributors). Most other projects do not have the resources to spare on unproven tech such as static analysis.

To make matters worse, at first people thought JavaScript was a toy language worth only cut’n'pasting from recipes online. Then just as JavaScript was getting more popular, SpiderMonkey embedding got buggier and made for some unpleasant first experiences with the Hydras.

Conclusion

There isn’t much to show for my work outside of Mozilla; that’s fine since my primary goal was Mozilla The Hydras aren’t dead, they are in maintenance mode.

I’m glad to see python-as-gcc-plugin approach, it seems to fill the same niche as Treehydra. I regret not starting out with Python (I think it’s slightly better than JavaScript for this task), I hope David Malcolm succeeds in attracting wider interest.

PS. I’m super-excited about the new DXR work. DXR is something that makes my daily life easier. DXR is by far the smartest code-indexing system out there, it’s bound to transform my life as a developer far more than any static analysis ever could

Telemetry went live in Firefox Nightly builds over the weekend. Everyone who wants to contribute to making Firefox better now has an easy new way: opt-in to telemetry.
There are two ways to opt in:
A) Click yes when prompted to report performance
B) Enable it by going to Options/Preferences, then Advanced/General tab

You can check on the data collected by installing my about:telemetry extension.

Telemetry is a way to gather stats about Firefox. Currently histograms are the main mechanism by which to gather data. There are 3 histogram types currently supported: exponential, linear and boolean. Exponential+linear histograms can accumulate numbers between 0 and a user-defined integer maximum, they differ in bucket size increments. Boolean histograms are meant to store 0 or 1.

Steps to Add a New Metric

1) Add a histogram definition to TelemetryHistograms.h specifying a histogram id, parameters and a description. For example HISTOGRAM(MYHGRAM, 1, 10000, 50, EXPONENTIAL, “Time (ms) taken by shiny new metric”) defines a MYHGRAM histogram with a minimum value of 1, maximum 10000 and 50 buckets that grow exponentially with the description specifying units as milliseconds.
2a) C++:

#include <mozilla/Telemetry.h>

….
Telemetry::Accumulate(Telemetry::MYHGRAM, <some interestin integer value goes here>)

2b) JS:

Telemetry = Cc["@mozilla.org/base/telemetry;1"].getService(Ci.nsITelemetry)
var h = Telemetry.getHistogramById("MYHGRAM");
h.Add(<some interesting value goes here>);

3) Install about:telemetry addon, go to about:telemetry to check that the chosen histogram type and parameters summarize your data in a useful way.

4) Commit and wait for results to come in.
FAQ
Q: What about those UMA_* macros?
A: Don’t use them. I added TelemetryHistograms.h so telemetry can be easily reviewed by security and privacy police. It also avoids a few pitfalls such as accidentally initializing the same histogram with different parameters, etc.

Q: When will telemetry be deployed?
A: We will land the UI to turn it on as soon as an updated privacy policy is posted. It should show up in nightly builds by the end of the week.

Q: How do I access the gathered telemetry data?
A: TBD. Data is stored on metrics server, we will figure this out soon.

Q: What about addons?
A: TBD. At the moment addons are free to inspect telemetry data in their browser. We haven’t decided on a process to let addon authors add new probes and access stats. For now, addons should not participate in telemetry.

My previous post was too optimistic. There will be no telemetry in Firefox 6. Due to the multitude of reviews involved we slipped and are now aiming for Firefox 7. Bug 659396 tracks various ongoing telemetry tasks.

I updated my about:telemetry extension to work with Firefox 7 nightlies. Additionally, my friend, David, helped me apply some nasty CSS tricks to make the histograms look like histograms. I’m open to further CSS contributions. I haven’t listed the extension on AMO because we plan to have this functionality integrated into Firefox soon (hopefully 7).

To turn on telemetry,we have to: a) finish up the telemetry opt-in UI and b) update our privacy policy.

Thanks to everybody who manually flipped the pref to turn on telemetry. Having early feedback on this feature is awesome.