MemShrink progress, week 20

Posted on November 2, 2011 by Nicholas Nethercote| 25 Comments

Surprise of the week

[Update: This analysis about livemarks may be wrong.  Talos results from early September show that the MaxHeaps number increased, and the reduction when the "Latest Headlines" livemark was removed has undone that increase.  So livemarks may not be responsible at all, it could just be a coincidence.  More investigation is needed!]

Jeff Muizelaar removed the “Latest Headlines” live bookmark from new profiles.  This was in the Bookmarks Toolbar, and so hasn’t been visible since Firefox 4.0, and most people don’t use it.  And it uses a non-zero amount of memory and CPU.  Just how non-zero was unclear until Marco Bonardo noticed some big performance improvements.  First, in the Talos “MaxHeap” results on WinNT5.2:

Talos MaxHeap graph

And secondly in the Talos “Allocs” results on WinNT5.2 and Mac10.5.2:

Talos Allocs graph

In the WinNT5.2 cases, it looks like we had a bi-modal distribution previously, and this patch changed things so that the higher of the two cases never occurred.  In the Mac10.5.2 case we just had a simple reduction in the number of allocations.  On Linux the results were less conclusive, but there may have been a similar if smaller effect.

This surprised me greatly.  I’ve done a lot of memory profiling of Firefox and never seen anything that pointed to the feed reader as using a lot of memory.  This may be because the feed reader’s usage is falling into a larger, innocuous bucket, such as JS or generic strings.  Or maybe I just missed the signs altogether.

Some conclusions and questions:

  • If you have live bookmarks in your bookmarks toolbar, remove them! [Update: I meant to say "unused live bookmarks".]
  • We need to work out what is going on with the feed reader, and optimize its memory usage.
  • Can we disable unused live bookmarks for existing users?

Apparently nobody really owns the feed reader, because previous contributors to it have all moved on.  So I’m planning to investigate, but I don’t know the first thing about it.  Any help would be welcome!

 Other stuff

There was a huge memory leak in the Video DownloadHelper add-on v4.9.5, and possibly earlier versions.  This has been fixed in v4.9.6a3 and the fix will make it into the final version v4.9.6 when it is released.  That’s one more add-on leak down, I wonder how many more there are to go.

TraceMonkey, the trace JIT, is no longer built by default.  This means it’s no longer used, and this saves both code and data space.  The old combination of TraceMonkey and JaegerMonkey is slower than the new combination of JaegerMonkey with type inference, and TraceMonkey is also preventing various clean-ups and simplifications, so it’ll be removed entirely soon.

I refined the JS memory reporters in about:memory to give more information about objects, shapes and property tables.

I avoided creating small property tables, removed KidHashes when possible, and reduced the size of KidHashes with few entries.

I wrote about various upcoming memory optimizations in the JS engine.

Justin Lebar enabled jemalloc on MacOS 10.5 builds.  This was expected to be a space improvement, but it also reduced the “Tp5 MozAfterPaint” page loading benchmark by 8%.

Robert O’Callahan avoided excessive memory usage in certain DOM animations on Windows.

Drew Willcoxon avoided excessive memory usage in context menu items created by the add-on SDK.

Bug Counts

  • P1: 35 (-1, +1)
  • P2: 116 (-2, +5)
  • P3: 55 (-2, +3)
  • Unprioritized: 5 (-4, +5)

At this week’s MemShrink meeting we only had 9 bugs to triage, which is the lowest we’ve had in a long time.  It feels like the MemShrink bug list is growing slower than in the past.

→ 25 Comments

Posted in Firefox, JägerMonkey, Memory consumption, MemShrink, Tracemonkey, Uncategorized

Reducing about:memory’s “heap-unclassified” measurement with DMD

Posted on November 1, 2011 by Nicholas Nethercote| Comments Off

about:memory is a really useful tool, but everyone always complains about the “heap-unclassified” number being too high.  It’s too high because we don’t have enough memory reporters implemented.

DMD is a tool that identifies where new memory reporters should be added to reduce “heap-unclassified”.  I’ve been using it for a couple of months now, and I’ve just written instructions on how to use it.  It’s not easy, but hopefully it’s doable.

It’s probably worth pointing out that we have a pretty good handle on what needs to be done to reduce “heap-unclassified” significantly — check out the list of memory reporters to be implemented.  Almost all of those bugs were filed based on data from DMD.  The single most important bug in that list is the one to add missing style reporters;  I see multiple megabytes of CSS stuff all the time in DMD’s output.

Comments Off

Posted in about:memory, DMD, Firefox, Valgrind

SpiderMonkey is on a diet

Posted on November 1, 2011 by Nicholas Nethercote| 20 Comments

One thing I’ve learnt while working for Mozilla is that a web browser can be characterized as a JavaScript execution environment that happens to have some multimedia capabilities.  In particular, if you look at Firefox’s about:memory page, the JS engine is very often the component responsible for consuming the most memory.

Consider the following snapshot from about:memory of the memory used by a single JavaScript compartment.

about:memory snapshot

(For those of you who have looked at about:memory before, some of those entries may look unfamiliar, because I landed a patch to refine the JS memory reporters late last week.)

There is work underway to reduce many of the entries in that snapshot.  SpiderMonkey is on a diet.

Objects

Objects are the primary data structure used in JS programs;  after all, it is an object-oriented language.  Inside SpiderMonkey, each object is represented by a JSObject, which holds basic information, and possibly a slots array, which holds the object’s properties. The memory consumption for all JSObjects is measured by the “gc-heap/objects/non-function” and “gc-heap/objects/function” entries in about:memory, and the slots arrays are measured by the “object-slots” entries.

The size of a non-function JSObject is currently 40 bytes on 32-bit platforms and 72 bytes on 64-bit platforms.  Brian Hackett is working to reduce that to 16 bytes and 32 bytes respectively. Function JSObjects are a little larger, being (internally) a sub-class of JSObject called JSFunction.  JSFunctions will therefore benefit from the shrinking of JSObject, and Brian is slimming down the function-specific parts as well.  In fact, these changes are complete in the JaegerMonkey repository, and will likely be merged into mozilla-central early in the Firefox 11 development period.

As for the slots arrays, they are currently arrays of “fatvals” A fatval is a 64-bit internal representation that can hold any JS value — number, object, string, whatever.  (See here for details, scroll down to “Mozilla’s New JavaScript Value Representation”;  the original blog entry is apparently no longer available).  64-bits per entry is overkill if you know, for example, that you have an array full entirely of integers that could fit into 32 bits.  Luke Wagner and Brian Hackett have been discussing a specialized representation to take advantage of such cases.  Variations on this idea have been tried twice before and failed, but perhaps SpiderMonkey’s new type inference support will provide the right infrastructure for it to happen.

Shapes

There are a number of data structures within SpiderMonkey dedicated to making object property accesses fast.  The most important of these are Shapes.  Each Shape corresponds to a particular property that is present in one or more JS objects.  Furthermore, Shapes are linked into linear sequences called “shape lineages”, which describe object layouts.  Some shape lineages are shared and live in “property trees”.  Other shape lineages are unshared and belong to a single JS object;  these are “in dictionary mode”.

The “shapes/tree” and “shapes/dict” entries in about:memory measure the memory consumption for all Shapes.  Shapes of both kinds are the same size;  currently they are 40 bytes on 32-bit platforms and 64 bytes on 64-bit platforms.  But Brian Hackett has also been taking a hatchet to Shape, reducing them to 24 bytes and 40 bytes respectively.  This has required the creation of a new auxiliary BaseShape type, but there should be many fewer BaseShapes than there are Shapes.  This change will also increase the number of Shapes, but should result in a space saving overall.

SpiderMonkey often has to search shape lineages, and for lineages that are hot it creates an auxiliary hash table, called a “property table”, that makes lookups faster.  The “shapes-extra/tree-tables” and “shapes-extra/dict-tables” entries in about:memory measure these tables.  Last Friday I landed a patch that avoids building these tables if they only have a few items in them;  in that case a linear search is just as good.  This reduced the amount of memory consumed by property tables by about 20%.

I mentioned that many Shapes are in property trees.  These are N-ary trees, but most Shapes in them have zero or one child;  only a small fraction have more than that, but the maximum N can be hundreds or even thousands.  So there’s a long-standing space optimization where each shape contains (via a union) a single Shape pointer which is used if it has zero or one child.  But if the number of children increases to 2 or more, this is changed into a pointer to a hash table, which contains pointers to the N children.  Until recently, if a Shape had a child deleted and that reduced the number of children from 2 to 1, it wouldn’t be converted from the hash form back to the single-pointer.  I changed this last Friday.  I also reduced the minimum size of these hash tables from 16 to 4, which saves a lot of space because most of them only have 2 or 3 entries.  These two changes together reduced the size of the “shapes-extra/tree-shape-kids” entry in about:memory by roughly 30–50%.

Scripts

Internally, a JSScript represents (more or less) the code of a JS function, including things like the internal bytecode that SpiderMonkey generates for it.  The memory used by JSScripts is measured by the “gc-heap/scripts” and “script-data” entries in about:memory.

Luke Wagner did some measurements recently that showed that most (70–80%) JSScripts created in the browser are never run.  In hindsight, this isn’t so surprising — many websites load libraries like jQuery but only use a fraction of the functions in those libraries.  It wouldn’t be easy, but if SpiderMonkey could be changed to generate bytecode for scripts lazily, it could reduce “script-data” memory usage by 60–70%, as well as shaving non-trivial amounts of time when rendering pages.

Trace JIT

TraceMonkey is SpiderMonkey’s original JIT compiler, which was introduced in Firefox 3.5.  Its memory consumption is measured by the “tjit-*” entries in about:memory.

With the improvements that type inference made to JaegerMonkey, TraceMonkey simply isn’t needed any more.  Furthermore, it’s a big hairball that few if any JS team members will be sad to say goodbye to.  (js/src/jstracer.cpp alone is over 17,000 lines and over half a megabyte of code!)

TraceMonkey was turned off for web content JS code when type inference landed.  And then it was turned off for chrome code.  And now it is not even built by default.  (The about:memory snapshot above was from a build just before it was turned off.)  And it will be removed entirely early in the Firefox 11 development period.

As well as saving memory for trace JIT code and data (including the wasteful ballast hack required to avoid OOM crashes in Nanojit, ugh), removing all that code will significantly shrink the size of Firefox’s code.  David Anderson told me the binary of the standalone JS shell is about 0.5MB smaller with the trace JIT removed.

Method JIT

JaegerMonkey is SpiderMonkey’s second JIT compiler, which was introduced in Firefox 4.0.  Its memory consumption is measured by the “mjit-code/*” and “mjit-data” entries in about:memory.

JaegerMonkey generates a lot of code.  This situation will hopefully improve with the introduction of IonMonkey, which is SpiderMonkey’s third JIT compiler.  IonMonkey is still in early development and won’t be integrated for some time, but it should generate code that is not only much faster, but much smaller.

GC HEAP

There is a great deal of work being done on the JS garbage collector, by Bill McCloskey, Chris Leary, Terrence Cole, and others.  I’ll just point out two long-term goals that should reduce memory consumption significantly.

First, the JS heap currently has a great deal of wasted space due to fragmentation, i.e. intermingling of used and unused memory.  Once moving GC — i.e. the ability to move things on the heap — is implemented, it will pave the way for a compacting GC, which is one that can move live things that are intermingled with unused memory into contiguous chunks of memory.  This is a challenging goal, especially given Firefox’s high level of interaction between JS and C++ code (because moving C++ objects is not feasible), but one that could result in very large savings, greatly reducing the “gc-heap/arena/unused” and “gc-heap-chunk-*-unused” measurements in about:memory.

Second, a moving GC is a prerequisite for a generational GC, which allocates new things in a small chunk of memory called a “nursery”.  The nursery is garbage-collected frequently (this is cheap because it’s small), and objects in the nursery that survive a collection are promoted to a “tenured generation”.  Generational GC is a win because in practice the majority of things allocated die quickly and are not promoted to the tenured generation.  This means the heap will grow more slowly.

Is that all?

It’s all I can think of right now.  If I’ve missed anything, please add details in the comments.

There’s an incredible amount of work being done on SpiderMonkey at the moment, and a lot of it will help reduce Firefox’s memory consumption.  I can’t wait to see what SpiderMonkey looks like in 6 months!

→ 20 Comments

Posted in about:memory, Firefox, Garbage Collection, JägerMonkey, Memory consumption, MemShrink, Tracemonkey

MemShrink progress, week 19

Posted on October 26, 2011 by Nicholas Nethercote| 11 Comments

Good News

Peter Van der Beken fixed a leak caused by chrome code that injects a function into pages.  This was a MemShrink P1 bug.  The commentary in the bug is confusing, but this may have been affecting numerous add-ons including Firebug. (Bug 669730 is open for tracking leaks in Firebug;  it hasn’t yet been confirmed whether this fix has helped Firebug.)

Andrew McCreight rewrote JS_DumpHeap so that it dumps the complete JS object graph.  This was a MemShrink P1 bug because it’s an important piece of infrastructure for writing leak detection tools.

Justin Lebar fixed the measurement of RSS on Mac.  This was a MemShrink P1 bug because it prevented us from enabling jemalloc on Mac 10.5 machines.

Brian Hackett tweaked the JS engine so that more method JIT code can be discarded periodically, particularly chrome code in system compartments.

Oleg Romashin avoided using transparent layers in Fennec remote offscreen viewports.  This saves 5MB of memory in the active tab.

I added a memory reporter for the startup cache.  This is often around 1MB of memory, but it’s allocated via mmap/VirtualAlloc and so doesn’t change the “heap-unclassified” number in about:memory.

Boris Zbarsky fixed a small leak involving CSS transforms.

Bad News

It appears a bad memory regression has occurred in the past week or so.  Several people have reported multi-second pauses caused by garbage collection and cycle collection.  The problem only seems to strike when many tabs are open and/or the browser has been running for multiple days.  This needs investigation;  if you are experiencing similar problems please report in the bug.  Reliable steps to reproduce this bug will be invaluable.  If you turn on javascript.options.mem.log in about:config you can see when GCs and CCs occur and how long they take, which is helpful for diagnosis.

Bug Counts

The current bug counts are as follows.

  • P1: 35 (-5, +3)
  • P2: 113 (-3, +8)
  • P3: 54 (-0, +1)
  • Unprioritized: 4 (-5, +3)

I want to point out this bug, which presents an idea to help hunt down reproducible leaks that occur when users have multiple add-ons present.  This is important because many of the leaks reported by users recently are due to add-ons, but often the reporter has many add-ons installed which makes finding the culprit painful.  The goal is to write a Firefox add-on that selectively disables installed add-ons, so that a user can bisect them to discover which add-on is responsible for the leak.  This bisecting process is something that people can do manually, but an add-on that automates the process would make things easier and less error-prone.  Mercurial’s ‘hg bisect’ command would serve as a useful comparison.  Ehsan Akhgari has volunteered to mentor anyone who would like to try to implement this.

 

→ 11 Comments

Posted in Fennec, Firefox, Garbage Collection, Memory consumption, MemShrink

MemShrink Progress, weeks 13–18

Posted on October 19, 2011 by Nicholas Nethercote| 55 Comments

I’ve been on vacation, so this report covers (incompletely) the past six weeks’ worth of MemShrink improvements.

Big Things

Paul Biggar and Mike Hommey enabled jemalloc on Mac 10.6.  This will hopefully reduce fragmentation on that platform.  And it brings it in line with Windows, Linux and Android.  Now we just need it for Mac 10.5 and Mac 10.7.

Oleg Romashin found a way to drop some Thebes layers in inactive tabs in Fennec.  I won’t pretend to understand the details of this bug, but if I’ve understood correctly it can saves 12MB or more per background tab.

Jeff Muizelaar turned on decode-on-draw.  This means that if you open a new page in a background tab, none of its images will be decoded until you switch to that tab.  Previously any images would be decoded and then discarded after 10 to 20 seconds (unless you switched to the tab before the time-out occurred).  This change can save a lot of memory (and CPU time) for users browsing image-heavy sites.

Gian-Carlo Pascutto optimized the safe browsing database.  This hopefully has fixed our url-classifier bloat problems.  (I plan to verify this soon.)

Chris Leary and Jonathan “Duke” Leto made regexp compilation lazy.  This potentially saves 10s or even 100s of MBs of memory in some cases by not compiling some regexps, and also allowing regexps to be GC’d more quickly.  There were some possible performance regressions from this patch, it’s unclear from the bug what exactly the state of these are.

Justin Lebar converted some uses of nsTArray to nsAutoTArray (and also here).  These avoided some calls to malloc (in the latter case, around 3% of all malloc calls!) and may help reduce heap fragmentation a little.  Robert O’Callahan did a separate, similar nsAutoTArray change here.  Justin also avoided another 1% of total malloc calls in JSAutoEnterCompartment.

Chris Leary rewrote JSArena, which avoided some wasted memory, as well as replacing some hoary old C code with sleek modern C++.

SMALLER THINGS

A new contributor named Jiten (a.k.a. deLta30) fixed about:memory’s GC and CC buttons so they trigger collections in both the content and chrome process in Fennec.  Previously only the chrome process was affected.  (I’m not sure how this change will be affected by the decision to switch to the native chrome process in Fennec.)  Great work, Jiten!

I avoided some wasted space in the JS code generator, and some more in the parser.  Justin Lebar did something similar in nsTArray_base.

Jonathan Kew added a memory reporter for textruns and associated data.  Justin Lebar added the “history-links-hashtable” memory reporter.

Justin Lebar fixed some bogus “canvas-2d-pixel-bytes” values in about:memory.

Brian Bondy fixed a leak in Windows drag and drop code.

Tim Taubert wrote about finding leaks in browser-chrome mochitests.

Bug Counts

The current bug counts are as follows.  The differences are against the MemShrink week 12 counts.

  • P1: 37 (-3, +11)
  • P2: 108 (-9, +37)
  • P3: 53 (-1, +14)
  • Unprioritized: 6 (-21, +5)

They’re still going up.  The good news is that my gut feeling is that not many of these bugs are problems reported by users.  (And those that are often are caused by add-ons.)  Most of the new reports are ideas for improvements from developers.

→ 55 Comments

Posted in Fennec, Firefox, Memory consumption, MemShrink

MemShrink progress, week 12

Posted on September 7, 2011 by Nicholas Nethercote| 35 Comments

about:memory improvements

Lots of changes were made to about:memory this week.

Justin Lebar landed a patch that provides detailed information about RSS, vsize and swap memory usage on Linux and Android.  (The patch was backed out due to a minor leak but I expect Justin will fix that and re-land it soon.)  This will help us understand memory usage that is not covered by the “Explicit Allocations” tree in about:memory, such as memory used for static code and data, and it should be particularly useful on Android.  The contents of the new trees are hidden by default;  you have to click on the tree heading to expand each one.

Kyle Huey split up about:memory’s layout measurements on a per-PresShell basis.  This makes it easy to see how much layout memory is being used by each web page.

Something I failed to mention last week was that with the landing of type inference, each JavaScript compartment has five new measurements: “object-main”, “script-main”, “tables”, “analysis-temporary”, and “object-empty-shapes”.

I converted some of the JavaScript memory reporters to use moz_malloc_usable_size to measure actual allocation sizes instead of requested allocation sizes.  This accounts for slop bytes caused by the heap allocator rounding up.  This is quite important — slop bytes can easily account for over 10% of the heap, and if we don’t account for them we’ll never get about:memory’s “heap-unclassified” number down.  Therefore I’ll be doing more of this in the future.  And it would be great if people writing new memory reporters can do the same thing!

Finally, on the topic of “heap-unclassified” number:  people often complain about it, so I’m happy to say that it is on a clear downward path.  Indeed, thanks to DMD, at the time of writing we have 16 open bugs to add new memory reporters for things that consume significant amounts of memory, and 13 of these are assigned.  I’m hoping that in a month or two the “heap-unclassified” number on development builds will typically be 10–15% rather than the 30–35% it usually is now.

Other things

I changed the growth strategy used for one of JaegerMonkey’s buffers to avoid large amounts of memory wasted due to slop bytes.  These buffers are short-lived so the fix doesn’t make a great difference to total memory consumption, but it does reduce the number of allocations and heap churn.

Marco Bonardo wrote about his recent changes to the handling of the places database.

Dietrich Ayala wrote about an experimental add-on that unloads tabs that haven’t been viewed in a while, which is an interesting idea.  I suspect the exact approach used in the add-on won’t be viable in the long run, but we can certainly benefit from doing a better job of discarding regenerable info that hasn’t been used in a while, particularly on mobile.

Bug counts

This weeks’s bug counts are as follows:

  • P1: 29 (-2, +2)
  • P2: 80 (-4, +8)
  • P3: 40 (-2, +4)
  • Unprioritized: 22 (-12, +12)

Just like last week, Marco Castelluccio tagged quite a lot of old bugs with “[MemShrink]“.  We had 45 unprioritized bugs at the start of this week’s meeting, and we got through more than 20 of them.

Some comments on last week’s post got me thinking about how to make it easier for more people to help with MemShrink.  For those who don’t have much coding experience, probably the best bet is to look at the list of unconfirmed bugs — these are problems reported by users where the particular problem hasn’t been identified.  Often they need additional effort to determine if they are reproducible, due to add-ons, etc.  For example, in bug 676872 a user was seeing very high memory usage, and it’s clear that it was caused by one or more of the 41(!) add-ons he had enabled.  Ideally that bug’s reporter would disable them selectively to narrow that down, but anyone could do likewise with some effort.

For those who do have coding experience, it would be worth looking at the list of bugs that have a “mentor” annotation.  For example, bug 472209 is about adding some graphing capability to about:memory.  Jezreel Ng made some excellent progress on this during his internship, it just needs someone to take over and finish it up.

Finally, for those who like a challenge or have some experience with Firefox’s code, the full list of unassigned bugs might be of interest.  There are currently 86 such bugs!  More than I’d like.

(BTW, I’ve added links for the three bug lists above to the MemShrink wiki page.)

On HIATUS

I will be on vacation for the next five weeks and the MemShrink progress report will be on hiatus during that time.  But MemShrink meetings will continue (except there won’t be one next week due to the Mozilla all-hands meeting).  I look forward to writing a bumper progress report on October 19, where I’ll be able to summarize everything that happened while I was away!

→ 35 Comments

Posted in about:memory, Firefox, Memory consumption, MemShrink

On vacation September 12 to October 17

Posted on September 7, 2011 by Nicholas Nethercote| 4 Comments

I will be on vacation from September 12 to October 17.  During that time I will be online sporadically, if at all.

I won’t be attending the Mozilla all-hands meeting.  My vacation was scheduled before the all-hands dates were announced, unfortunately.

However, I will visit Mozilla HQ in Mountain View on September 27 and 28.  This means I’ll be present for the launch of Firefox 7 (the first release to benefit from MemShrink) and also for a MemShrink weekly meeting.

→ 4 Comments

Posted in Personal

MemShrink progress, week 11

Posted on August 31, 2011 by Nicholas Nethercote| 8 Comments

This week was quiet in terms of patches landed.

  • Marco Bonardo changed the way the places.sqlite database is handled. I’m reluctant to describe the change in much detail because I’ll probably get something wrong, and Marco told me he’s planning to write a blog post about it soon.  So I’ll just quote from the bug: “Globally on my system (8GBs) I’ve often seen places.sqlite cache going over 100MB, with the patch I plan to force a maximum of 60MB (remember this will vary based on hardware specs), that is a >40% improvement. We may further reduce in future but better being on the safe side for now.”  This was a MemShrink:P1 bug.
  • New contributor Sander van Veen knocked off another bug (with help from his friend Bas Weelinck) when he added more detail to the “mjit-code” entries in about:memory.  This makes it clear how much of JaegerMonkey’s code memory usage is for normal methods vs. memory for compiled regular expressions.
  • I rearranged nsCSSCompressedDataBlock to avoid some unnecessary padding on 64-bit platforms.  This can save a megabyte or two if you have several CSS-heavy (e.g. Gmail) tabs open.   It makes no difference on 32-bit platforms.

But it was a very busy week in terms of bug activity.  Let’s look at the numbers.

  • P1: 29 (-2, +2)
  • P2: 76 (-10, +20)
  • P3: 38 (-1, +2)
  • Unprioritized: 22 (-5, +23)

Several things happened here.

  • Marco Castelluccio looked through old bugs and found a lot (30 or more) that were related to memory usage and tagged them with “MemShrink”.
  • Nine new bugs were filed to reduce about:memory’s “heap-unclassified” number by adding memory reporters;  many of these were thanks to Boris Zbarsky’s insights into the output produced by DMD.
  • I closed out a number of bugs that were incomplete, stale, or finished;  this included some of those newly marked by Marco, and some ones that were already tagged with “MemShrink”.
  • I tagged five leaks that were found with the cppcheck static analysis tool.

We spent the entire MemShrink meeting today triaging unprioritized bugs and we got through 23 of them.  Of the remaining unprioritized bugs, the older ones tagged by Marco and the cppcheck ones (which I tagged after the meeting) constitute most of them.

It’s clear that the rate of problem/improvement identification is outstripping the rate of fixes.  We have a standing agenda item in MemShrink meetings to go through Steve Fink’s ideas list, but we haven’t touched it in the past two meetings because we’ve spent the entire time on triage.  And when we do go through that list, it will only result in more bugs being filed.  I’m hoping that this glut of MemShrink-tagged bugs is temporary and the new bug rate will slow again in the coming weeks.

In the meantime, if you want to help, please look through the lists of open bugs, or contact me if you aren’t sure where to start, and I’ll do my best to find something you can work on.  Thanks!

→ 8 Comments

Posted in about:memory, Firefox, JägerMonkey, Memory consumption, MemShrink, SQLite

“Browser X is using Y MB of memory with Z tabs open” is a meaningless observation

Posted on August 29, 2011 by Nicholas Nethercote| 23 Comments

I read an exchange today about the memory usage of Firefox 7 (which is currently in beta).

It uses 600 MB of RAM with 10 tabs open (different websites).

lol wut? I’m not getting the same results as you. I have 12 tabs open here… RAM usage rarely exceeded 300mb for me. That’s already much better than Chrome, which seems to be using 190mb for just 2 tabs.

Memory consumption is around 300 MB for me too.

I’ve seen variants of this exchange countless times.  Unfortunately, observations such as “browser X is using Y MB of memory with Z tabs open” are meaningless.  To understand why, consider this alternative, hypothetical exchange:

I ran 3 different programs on my computer, they took 55 seconds to finish.

lol wut?  I just ran 8 programs and they only took 20 seconds to finish!

You’d never see an exchange like that, because it’s obvious that a comparison of the run-time of multiple programs is meaningless if you don’t specify what those programs are.  Workload matters. And yet, people make comparisons of browser memory usage like this all the time.

Here’s a simple example to show why this isn’t meaningful.  I just did some memory measurements with a development build of Firefox 9, using a moderately-used profile with no add-ons installed.  First I started it and opened 10 instances of www.google.com.au, one per tab, and the resident set size (RSS) was 139MB.  Then I re-started it and opened 10 instances of gmail.com, and the RSS climbed to 651MB.

This is hardly surprising.  gmail.com is a fully-fledged email client.  www.google.com.au is not much more than a search box.

Why do people so often make these meaningless “Y MB of memory with Z tabs open” comments?  My theory is that it’s because most people have no idea how complicated web browsers and web pages are.  Mild experience with HTML authoring is endemic — heaps of people have thrown together a basic web page, and it’s just a document with with some structure and styling, right?  So they think the difference between the website they wrote for their scout troop in 1997 (the one with the “under construction” animated GIF) and the front page of TechCrunch is merely a difference of degree, not kind.

They’re wrong.  For one, HTML and related technologies are hugely more powerful and complicated now than they were a few years ago.  Also, their scout troop website probably didn’t contain multiple megabytes of JavaScript code tracking its visitors’ every move. A web browser is not a document viewer, it is a full-blown programming environment with some very sophisticated text and graphical capabilities.  A web page is not a document but a program.  Therefore, the memory (and CPU) usage of different web pages varies dramatically.

Here’s a check-list of information that you should include if you want an observation about browser memory usage to be meaningful.

  • What sites do you have open?  The more specific, the better.  E.g. “Gmail and a few nytimes.com pages” is ok, but listing the exact URLs is better.
  • How did the browser get in this state?  Did you just start it, or have you been using it and visiting other sites for hours?
  • What memory metric are you using, and how did you measure it?  “Explicit” from Firefox’s about:memory?  “Private bytes” from the Windows Task Manager?  “RSS” from “top” on Mac or Linux?  (The full output of Firefox’s about:memory is hugely useful, because it includes dozens of highly relevant measurements.)
  • In Firefox, do you have any add-ons installed?  That can (and often does) make a big difference.  (And that’s a topic for another day.)

If you include all that, it’s highly likely that somebody else can reproduce your measurements, which means it’s a highly meaningful observation.

On a related note, I wrote previously about Firefox 7′s memory improvements, saying “Firefox 7 uses less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.”  The presence of the words “often” and “sometimes” in that sentence were deliberate.  We’ve seen those numbers in our testing, but once again, workload matters.  I really hope the numbers we’ve seen match what normal users see.  In fact, I thought that people would independently test these claims shortly after I wrote them.  But, to my knowledge, that hasn’t happened so far, even though those claimed improvements have been reported far and wide.  (I’ve even seen numerous headlines that say “Firefox 7 to use 50% less memory”, alas.)  Maybe once Firefox 7 is officially released people will make independent measurements.

(A final note:  suspicious readers may think that I’m trying to obliquely absolve Firefox from any responsibility for its memory usage by blaming web pages instead.  I’m not.  Firefox is doing much better now, but there’s still plenty of ways it can be improved;  please contact me if you want to help.)

→ 23 Comments

Posted in Firefox, Memory consumption

MemShrink progress, week 10

Posted on August 24, 2011 by Nicholas Nethercote| 17 Comments

A quieter week this week.  Well, plenty of work was done but not yet completed, and I mostly write only about changes that have been finished.

Now for the bug counts.  (Canned MemShrink bug searches are available here.)

  • P1: 29 (-5, +4)
  • P2: 66 (-6, +8)
  • P3: 37 (-1, +2)
  • Unprioritized: 4 (-1, +4)

There was lots of P1 movement, which is good:  a couple were fixed, some have had enough progress made on them that they were able to be downgraded to P2, and some new problems/opportunities were identified.

 

→ 17 Comments

Posted in Firefox, Memory consumption, MemShrink