The thing to watch when you’re measuring software security, though, is that you’re measuring the things that matter. We’ve talked about this before, but it bears repeating: if you measure the wrong things, you encourage vendors to game the system instead of actually making things better.

What Makes A Good Security Metric?

There isn’t a single statistic you can gather that will give you a complete picture of security. Any robust security metrics model will need to take into account multiple factors. Nevertheless, there are 3 essential elements that should underlie any well-designed model. We call them the SEC essentials:

Severity : A good measurement model will put more emphasis on severe, automatically exploitable bugs than it does on nuisance bugs or ones that require users to cooperate extensively with their attacker. Measuring severity encourages vendors to fix the right bugs first, not to pad their numbers with minor fixes while major vulnerabilities languish.

Exposure Window : It’s not very informative to count the absolute number of bugs but it is very important to know how long each bug puts users at risk. Measuring exposure window encourages vendors to fix holes quickly, and to get those fixes out to users.

Complete Disclosure : The other measurements you compile are almost meaningless if you can’t see all the fixed bugs. Some vendors only disclose flaws found by outside sources, concealing those discovered by their internal security teams to keep their bug numbers down. Measuring only externally-discovered vulnerabilities rewards vendors who are purely reactive and, worse, it fails to credit vendors who develop strong internal security teams. Those teams often find the majority of security bugs; it’s important that any security metric recognizes and encourages it.

What’s The Solution?

If it were easy to find a calculation that included all of this information in a universal way, we’d be using it. When we wrote about our metrics project last year, it was with the aim to develop these ideas, and to change the tone of the discussion.

If the work there has taught us anything, it is that this will not happen overnight. The first step, though, is being clear about what we should expect from any assessment of security. If it doesn’t focus on the three SEC essentials: Severity, Exposure Window, and Complete Disclosure, ask yourself why not. And then ask the people doing the measuring.

Johnathan Nightingale
Human Shield

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security
Bulletins.

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.

The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.

Lucas Adamski
Director of Security Engineering

Counting security vulnerabilities to compare the security of different software projects is flawed.  It is only a useful metric if you are comparing a project to itself over time.  I’ve discussed this topic here and here.  It’s even more ridiculous to try and compare an open source bug count to a closed source project because you can see all the bugs in an open source project.  You can only see the publicly found security issues for a closed source product, like Internet Explorer.

So what is interesting in the Techworld article is the measures of real risk to users:

“‘[Z]ero-day’ security bugs in Firefox were patched more quickly than in Microsoft Internet Explorer…”

“[I]n an examination of zero-day flaws – reported by third parties before a patch was available – Secunia found that Firefox tended to get more patches, sooner, compared to IE.”

“Out of eight zero-day bugs reported for Firefox in 2007, five have been patched, three of those in just over a week. Out of 10 zero-day IE bugs, only three were patched and the shortest patch time was 85 days.”

At Mozilla we work as hard as we can to ship fixes as soon as possible to minimize the exposure to our users.  It is great to see that the efforts we are making to minimize risk to users are paying off.

Since all software has bugs, it’s more important to consider how long it takes to get a fix out when a security issue is discovered than it is to count bugs. Number of vulnerabilities identified is a function of how many bugs are present, but is probably more influenced by things like who is looking, and how good they are at finding security issues. That makes it a misleading metric.

When you compare how long it takes Microsoft to fix Internet Explorer vulnerabilities versus how long it takes Mozilla to fix vulnerabilities in Firefox it becomes clear why he chose to count vulnerabilities in this report instead. Earlier this year Brian Krebs of the Washington Post came up with this when comparing IE to Firefox:

For a total 284 days in 2006 (or more than nine months out of the year), exploit code for known, unpatched critical flaws in pre-IE7 versions of the browser was publicly available on the Internet…

In contrast, Internet Explorer’s closest competitor in terms of market share — Mozilla’s Firefox browser — experienced a single period lasting just nine days last year in which exploit code for a serious security hole was posted online before Mozilla shipped a patch to remedy the problem.

Mike Schroepfer goes into this in more detail in his post today.

One of the goals of the bug counting report is to demonstrate that Microsoft fixed fewer bugs for IE than Mozilla did for Firefox. Unfortunately for Microsoft (and for anyone trying to use this report as analysis of useful metrics) he does not count all the security issues. If he were able to count them all, Microsoft could get credit for all the bugs they fixed. He counts only the public issues, because that is all Microsoft will tell us about. Microsoft is worried that if it ever says it has fixed X security issues, the world will focus on that it had X vulnerabilities in the first place, not that they are now fixed and no longer a risk for users. So the set of issues that are available for public comparison is limited to the set of vulnerabilities that are reported externally AND fixed in security updates.

This is a small subset of all the vulnerabilities, because the vulnerabilities that are found through the QA process and the vulnerabilities that are found by the security folks they engage as contractors to perform penetration testing are fixed in service packs and major updates. For Microsoft this makes sense because these fixes get the benefit of a full test pass which is much more robust for a service pack or major release than it is for a security update.

Unfortunately for Microsoft’s users this means they have to wait sometimes a year or more to get the benefit of this work. That’s a lot of time for an attacker to identify the same issue and exploit it to hurt users. Sometimes it just takes time to put in a complicated fix. Anyone that has shipped a major piece of software can relate to that. But this is not the case for every internally found security issue. Extending this process to include fixes that are ready and just sitting on the tree waiting for the preferred vehicle to ship increases risk for users. But it sure keeps those bug count numbers down.

If we as an industry would just acknowledge that counting bugs is useless then vendors could feel safe talking about what they are doing to protect users. At Mozilla we fix our bugs openly. When you count Mozilla security bugs you are seeing not just those that are reported externally, but also the ones that would be considered internal if we acted like most other software vendors. Since all software has security vulnerabilities, we consider a vulnerability identified and fixed a win. It speaks to the strength of our community based security efforts to actively identify and quickly fix security issues. We don’t let fixes languish on the tree waiting for a major release while users are vulnerable. We ship fixes regularly because securing our users is more important than protecting our PR team from having to respond to articles about counting bugs instead of looking at the metrics that actually indicate whether a vendor is doing reasonable things to keep users secure.

We’re not building fixes for our PR team, we’re building them for our users. Go ahead and count.

We spend a lot of time thinking about how we can get fixes out faster to users. But the window of risk is actually determined by two factors. The first is the time it takes us to create a patch, let call this Time to Fix. This includes the time to investigate a security issue, develop and test a fix, and finally ship the update. This is a better measure for understanding how safe a user is going to be than simply counting bugs.

But there’s another aspect to getting the fix to the user that often goes overlooked. That is the Time to Deploy. Time to Deploy is how long it takes for users to get a patch installed once the fix is available from the vendor. Auto-update has gone a long way toward minimizing Time to Deploy for Firefox, but there are still areas on which we can improve.

This chart shows how long it took for users to move from 1.5.0.5 to 1.5.0.6 last year:

This shows that it took eight days for about 90 percent of Firefox users to get updated. When I saw this last year I thought it was pretty fantastic. Firefox has millions and millions of users. Getting almost everyone updated in just eight days seemed pretty incredible to me.

I ran the numbers again this year after we shipped 2.0.0.4.

This chart shows how long it took for users to move from 2.0.0.3 to 2.0.04 last month:

This time it only took six days to update 90 percent of users. That’s a 25 percent decrease in Time to Deploy and a significant improvement in reducing the window of opportunity for attackers to take advantage of security vulnerabilities. It appears that some of the improvements in infrastructure have contributed to these numbers so a big thank you to everyone working in IT and to our partners that host mirrors. You’re helping to keep Firefox users safe.