Security Metrics that Matter
April 15th, 2008 by Asa
A number of press articles surrounding Symantec’s Internet Security Threat Report, and other recent similar reports from Cenzic and Secunia, are offering the confusing and incorrect conclusion that the effective security and safety of web browsers can be measured by simply counting the number of vendor disclosed software flaws.
At Mozilla, we’re concerned that this flawed measure of security does a disservice to users trying to navigate the already tricky Web security landscape.
This kind of measuring is flawed for several reasons, all related in that they make it more difficult for consumers to make informed decisions about their online safety.
First, for complex programs like Web browsers, the number of vulnerabilities identified is often more influenced by things like who is looking, and how good they are at finding security issues, than the number of vulnerabilities actually present in the software.
Second, not all security flaws have the same impact on user safety. Comparing flaws which could allow arbitrary code execution to denial of service susceptibility doesn’t really provide users with actionable information.
Third, different organizations have different disclosure policies so a reporter or a user cannot get useful information by comparing the bug counts for products from different vendors.
And this is actually the fatal flaw of this kind of bug counting. When you count Mozilla security bugs you are seeing not just those flaws found and reported by third-party security researchers, but also the ones discovered and reported by Mozilla people — bugs that would be considered internal and probably never disclosed if we acted like most other software vendors.
That’s simply not the way other browser vendors disclose flaws. For those vendors, only flaws discovered by outside security researchers are disclosed, under pressure from the security researchers who would go public if the vendor did not. Internally found flaws are fixed and deployed, often in long delayed service packs, without the same disclosure.
This makes comparing the counts for those vendors and the counts available from Mozilla a true apples and oranges situation.
All of that brings up the very legitimate question, how should security be measured and how can people use better measures to make better choices.
For more than a year, Mozilla, along with members of the broader security community and some in the security tech press, has been advocating for measures of security that are both simple to digest, and much more useful to people trying to make secure technology decisions.
One measure we’ve settled on is called “Window of Exposure” and it is measured as the difference in days between the time at which exploit code affecting a vulnerability is made public and the time at which the affected vendor makes a patch available to the public for that vulnerability.
A variant of this measure is included in the Symantec report. This is great, but unfortunately it’s buried late in the report. Because simple bug counting is so much easier to explain, the better measure often gets left out of press accounts entirely.
A second measure is is called “Time to Deploy”. Time to Deploy is how long it takes for users to get a patch installed once the fix is available from the vendor. Just because it’s fixed, doesn’t mean that users all have the fix, leaving them vulnerable. The most important factor in minimizing Time to Deploy is the efficacy of the vendor’s automatic update system.
These two combined measures describe in fairly simple terms the number of days that a particular vendor’s users are vulnerable to security threats that result from software flaws. It’s our hope that more people will adopt these measures and that as a result, users will be able to make much more informed technology decisions.
So, we’d encourage Symantec to shift the emphasis to the more meaningful metric they already report, to begin measuring and reporting on Time to Deploy, and to and stop using the flawed apples to oranges bug count altogether.
And one final note: There are a lot of web browser advocates who will jump on any new report that compares browser security as an opportunity to promote their favorite. This happens all over the Web, and certainly within the Mozilla community and the communities of the other popular browsers.
Members of the Mozilla community are encouraged, when talking about browser security, to push back against meaningless bug counts, even when those counts would favor Mozilla. Instead, they should focus on measures that truly reflect how safe the browsers are, like the aforementioned “Window of Exposure” and “Time to Deploy.”
More reading
Mozilla Developer News » Better Metrics for Security - Understanding the Symantec Internet Security Threat Report
Mozilla Security Blog » Read past the headlines - Firefox is fixed faster
Mozilla Security Blog » Time to Deploy improvement of 25 percent
Mozilla Security Blog » Critical Vulnerability in Microsoft Metrics
Around the Web
Symantec Internet Security Threat Report:
Symantec Corporation » Internet Security Threat Report, Symantec Corporation » Internet Security Threat Report Volume XIII: April, 2008 (PDF), Mozilla Links » Handle with care: Symantec on web browsers security, Ars Technica » Report: Microsoft fastest to issue OS patches, Sun slowest, CIO » Report: ActiveX, QuickTime are buggiest browser plug-ins, Times Online » Number of computer viruses tops one million, ReadersZone » Most Insecure Browser Mozilla Firefox, CNET News.com’ One More Thing » Mac security not so much about the Mac, Builder AU News » Malware writers now number one software makers, Softpedia » Browser Wars: Internet Explorer vs. Firefox. vs. Safari vs. Opera - Vs. vulnerabilities in 2007, ComputerWorld - Security » Research fingers ActiveX, QuickTime as buggiest browser plug-ins
Cenzic Application Security Trend Report:
Cenzic » Application Security Trend Report Highlights 2007 as Another Crisis Year for Web Security, InfoWorld’s Security Watch » Pervasive Web apps flaws under siege, Redmond Developer News » Study: The Year’s Top-10 Web Application Vulnerabilities, Softpedia » Is Internet Explorer Safer Than Firefox, Opera and Safari? - Just because it has less security vulnerabilities?, Secure Web » 2007 - Another crisis year for Web Security!, Windows Vista Magazine » Internet Explorer 7 the safest browser ever, Help Net Security (HNS) » Microsoft Internet Explorer least vulnerable browser in Q4
Secunia and Microsoft’s Jeff Jones’ Internet Explorer and Firefox Vulnerability Analysis:
Jeff Jones Security Blog » Internet Explorer and Firefox Vulnerability Analysis, PC World » Red Hat and Firefox More Buggy Than Microsoft Apps, coldtobi’s blog » Apples and Oranges, Firefox and Internet-Explorer, Ars Technica » Mozilla scoffs at vulnerability study rating IE superior to Firefox, ComputerWorld » Microsoft, Mozilla trade punches over browser security, InfoWorld » Vulnerability counts do matter
Other:
Mark J Cox » Transparency, EWeek » Microsoft Patches: When Silence Isn’t Golden
Legal action agains symantec is in order.
Here is what Symantec’s report said about the security implications of the vulnerabilities:
“The growth in browser market-share for browsers such as Mozilla Firefox is a driving factor in the increased attention by security researchers. However, this does not necessarily result in more attack activity in the wild. Although internet explorer was subject to fewer vulnerabilities that are inherent to the browser in comparison to Mozilla, exploit activity in the wild indicates that it is still the gateway for third-party vulnerabilities affecting ActiveX and other browser plug-in technologies.”
Dear Asa,
I work for a financial institution where we use the following metric for app security: We simply add the days all known bugs stay unpatched. This gives us a figure that say how we are, if we are getting better or worse and if we are doing better or worse than other financial institutions.
I think the same principle could be used to compare Mozilla and other web browsers, you would get a metric that wouldn’t say what browser is safest, but what browser has the best security stance TODAY.
vaceituno: I assume you mean security bugs, this is kind of a good idea, but it’s also wise to take in to account the severity of the bug. I’ve seen security bugs be ignored by vendors because there’s no proof of concept code and it’s hard to think up of one, so it’ll likely never be exploited.
This is sometimes good and sometimes bad, some bugs do seem so minor and would require so much work that developers wouldn’t be wise to spend time there. That said I do often see Mozilla fix a lot of security bugs with no Proof of Concept code, especially if it’s memory related, and Microsoft just say “This is not a security bug”.