How Mozilla finds crash bugs

07.20.09 - 05:36pm

This Tuesday (2009-07-21), I’m organizing a crash bug triage day where anyone interested can help us classify the swamp of open crash bugs. Join us in #bugday on irc.mozilla.org if you’d like to help.

Crashes and security

Some Firefox crash bugs are severe security bugs. A crash bug is likely to be exploitable if it can be triggered by a web page and the bug is a memory safety bug such as calling a virtual method on a dangling pointer.

Although only a fraction of our crashes are exploitable, two thirds of our most severe security bugs are crashes. We’re striving to improve how we find and fix crash bugs, since the better we can find and fix the bugs, the more stable and secure Firefox will be.

Bug reports

When a user reports a bug, a loose team of volunteers tries to reproduce the bug, improve the bug report, and inform the correct developers about the bug.

The history of bug 503286 demonstrates the variety of skills involved in fixing crash bugs. Soon after zbyte filed the bug, Ria Klaassen reproduced the bug on her machine, captured a stack trace, and determined when the bug had been introduced. Volunteers Lucas and Natch reduced the web page to a simple testcase that demonstrated the bug. Using this information, JavaScript engine developers Andreas Gal and Blake Kaplan had little trouble creating a patch. All of this happened within 14 hours of the bug being reported.

Our current bug triage process is not perfect, and sometimes leaves valid bugs unprocessed. During this Tuesday’s crash bug triage day, we will experiment with a new triage workflow that should be both more efficient and more effective. If you’d like to help clear the crash bug backlog, please join us in #bugday this Tuesday (2009-07-21).

Reporting bugs requires substantial effort from users, and requires both English-language and technical skills, so we prefer not to rely exclusively on user-reported bugs. Luckily, crashes are easier to detect automatically than most other types of bugs, so we can find many of them using other methods.

Crash reports

Whenever Firefox crashes, a dialog appears that allows users to submit information about the crash to Mozilla. Although the average Firefox user only sends 1.5 crash reports per year, this information is valuable in aggregate.

Currently, our main use for these crash reports is to identify the most common crashes. If a crash is common but has not been reported in Bugzilla, we can look at the comments that come with some crash reports to get an idea of what triggers the crash.

Fixing common crashes is enough to make Firefox stable for most users, but even a rare crash could be a security hole, so we need to do more. To address this issue, Mozilla is creating a “crash reproduction farm” of computers that will automatically load URLs that come with crash reports. This will let us identify and fix most crashes that result from simply loading a web page.

Fuzz testing

Fuzz testing is the art of creating random input for software in order to find bugs. We do extensive fuzz testing of our most complex components, such as the JavaScript engine. Fuzz testing finds two thirds of the exploitable crashes we fix, including many that are too obscure for normal web sites to trigger.

Fixing crash bugs

For developers who are familiar with the relevant code, crashes are often easier to debug and fix than they are to find. To identify the developers familiar with the code, you can click the source code links in crash reports and see who last touched each line of code.

Jesse Ruderman
Security bug hunter

Category: Firefox, Security | | 6 Comments »

milw0rm 9158 “stack overflow” crash not exploitable (CVE-2009-2479)

07.19.09 - 02:44pm

In the last few days, there have been several reports (including one via SANS) of a bug in Firefox related to handling of certain very long Unicode strings. While these strings can result in crashes of some versions of Firefox, the reports by press and various security agencies have incorrectly indicated that this is an exploitable bug. Our analysis indicates that it is not, and we have seen no example of exploitability.

Details

On Windows, Firefox 3.0.x and Firefox 3.5.x are terminated due to an uncaught exception during an attempt to allocate a very large string buffer; this termination is safe and immediate, and does not permit the execution of attacker code.

On the Macintosh in Firefox 3.0.x and 3.5.x, a crash occurs inside the ATSUI system library (part of OS X), due to what appears to be a failure to check allocation results. This issue is likely to affect any application using the recommended text-handling libraries on OS X. We have reported this issue to Apple, but in the event that they do not provide a fix we will look to implement mitigations in Mozilla code. We recommend that other developers who use these libraries consider a similar practice, and we have added mitigations in the past for similar bugs in these libraries.

On Linux, the problem is similar to that on Mac: there is an abort in system libraries (pango, glib, libc). Due to the wide variation of Linux libraries and versions deployed, and different compilation options chosen by Linux distributors for Firefox, the details of the crash report may vary between machines.

As a result of our analysis, we do not believe that this represents an exploitable vulnerability in Firefox. Further, we believe that the IBM report is in error, and that the severity rating in the National Vulnerability Database report is incorrect. We have contacted them and hope to resolve the inaccuracies shortly.

[Updated (July 19, 8:50pm EDT): thanks to Larry Seltzer for bringing to our attention that Firefox 3.5.x will indeed still crash using the provided PoC on Windows, at least for some users.]

[Updated (July 20, 8:50am EDT): the SecurityFocus report has been updated to indicate that it is only a denial of service issue. This is consistent with our analysis; thanks to SecurityFocus for correcting their error.]

[Updated (July 20, 9:15am EDT): added results for Linux, thanks to Kevin Brosnan.]

Mike Shaver
VP Engineering, Mozilla Corporation

Category: Firefox, Security, Vulnerabilities | | 16 Comments »

Critical JavaScript vulnerability in Firefox 3.5

07.14.09 - 10:15am

Issue

A bug discovered last week in Firefox 3.5’s Just-in-time (JIT) JavaScript compiler was disclosed publicly yesterday. It is a critical vulnerability that can be used to execute malicious code.

Impact

The vulnerability can be exploited by an attacker who tricks a victim into viewing a malicious Web page containing the exploit code. The vulnerability can be mitigated by disabling the JIT in the JavaScript engine. To do so:

  1. Enter about:config in the browser’s location bar.
  2. Type jit in the Filter box at the top of the config editor.
  3. Double-click the line containing javascript.options.jit.content setting the value to false.

Note that disabling the JIT will result in decreased JavaScript performance and is only recommended as a temporary security measure.  Once users have been received the security update containing the fix for this issue, they should restore the JIT setting to true by:

  1. Enter about:config in the browser’s location bar.
  2. Type jit in the Filter box at the top of the config editor.
  3. Double-click the line containing javascript.options.jit.content setting the value to true.

Alternatively, users can disable the JIT by running Firefox in Safe Mode.  Windows users can do so by selecting Mozilla Firefox (Safe Mode) from the Mozilla Firefox folder.

Status

Mozilla developers are working on a fix for this issue and a Firefox security update will be sent out as soon as the fix is completed and tested.

Credit

Zbyte reported this issue to Mozilla and Lucas Kruijswijk helped reduce the exploit test case.

Update: This vulnerability has been fixed in Firefox 3.5.1, released Thursday, July 16, 2009

Category: Firefox, Security, Vulnerabilities | | 80 Comments »

Shutting Down XSS with Content Security Policy

06.19.09 - 03:41pm

For several years, Cross-Site Scripting (XSS) attacks have plagued many of the web’s most popular sites and victimized their users. At Mozilla, we’ve been working for the last year on a new technology called Content Security Policy, designed to shut these attacks down. We wanted to give a bit of background on this project as well as provide an update on our progress so far.

XSS is possible because all the content received as part of a web server response is treated with equal privilege by the requesting browser. JavaScript and other content included in a web page are all combined into a single security context which has full access to the DOM. Content Security Policy (CSP) provides a mechanism for sites to explicitly tell the browser which content is legitimate. The browser can then disregard any content which has not been blessed by the site.

In order to differentiate legitimate content from injected or modified content, CSP requires that all JavaScript for a page be 1) loaded from an external file, and 2) served from an explicitly approved host. This means that all inline script, javascript: URIs, and event-handling HTML attributes will be ignored. Only script included via a <script> tag pointing to a white-listed host will be treated as valid. Additionally, CSP allows several other common-sense security restrictions to be enforced.

We realize that this model is dramatically different than the current unrestricted model for the Web. We offer the following case supporting CSP’s adoption:

  1. CSP can be implemented in phases.

    While the biggest security benefit offered by Content Security Policy is the mitigation of XSS through inline script blocking, the migration of all JavaScript to external files may be challenging or time-consuming for some sites. Therefore, sites may choose to use the other features of Content Security Policy without adopting the JavaScript restrictions. Our hope is that this flexibility will provide a wide gate for such sites to adopt CSP in a limited fashion early, and later move toward a full implementation as time and resources permit.

  2. Even complex sites can be modified to support CSP.

    We have looked at HTML/JavaScript samples from a wide variety of websites ranging in complexity and have yet to see an example which could not be modified to support CSP. We’ll provide documentation regarding best practices for migrating a site to use CSP. Content Security Policy is also consistent with the programming paradigm “don’t mix code with content” so there may be additional functional benefits to be gained by implementing such separation.

  3. Drive a stake through the heart of XSS!

    XSS vulnerabilities have real value to attackers and are shared rapidly across the Web once discovered. Sites can breathe a little easier knowing that their users are protected, even if a XSS bug slips through. Because CSP can be configured to notify the protected site when an attack is blocked, CSP will even benefit users of older browsers, by helping sites identify and plug vulnerabilities quickly. The bottom line is that it will be extremely difficult to mount a successful XSS attack against a site with CSP enabled. All common vectors for script injection will no longer work and the bar for a successful attack is placed much, much higher.

Content Security Policy has been a collaboration of many individuals and has received input from multiple web sites, browser vendors, and web app security researchers. We are very excited to have reached a level of stability in the design that has allowed us to begin implementation of the CSP specification. Stay tuned for further updates. We will let you know when the fixes have been checked in to trunk and the product is ready to be tested in our nightly builds. Let us know what you think!

Brandon Sterne
Security Program Manager

Category: Firefox, Security | | 28 Comments »

Measure What Matters – The SEC Essentials

04.22.09 - 11:06am

People want to know that they are safe when they browse the web. There are important differences between browsers when it comes to security, and so it’s no surprise to see a growing number of groups out there attempting to compare browsers based on their security record. That’s great news; not only does it help inform users, but it also lets browser authors know where they stand, and where they can improve.

The thing to watch when you’re measuring software security, though, is that you’re measuring the things that matter. We’ve talked about this before, but it bears repeating: if you measure the wrong things, you encourage vendors to game the system instead of actually making things better.

What Makes A Good Security Metric?

There isn’t a single statistic you can gather that will give you a complete picture of security. Any robust security metrics model will need to take into account multiple factors. Nevertheless, there are 3 essential elements that should underlie any well-designed model. We call them the SEC essentials:

Severity : A good measurement model will put more emphasis on severe, automatically exploitable bugs than it does on nuisance bugs or ones that require users to cooperate extensively with their attacker. Measuring severity encourages vendors to fix the right bugs first, not to pad their numbers with minor fixes while major vulnerabilities languish.

Exposure Window : It’s not very informative to count the absolute number of bugs but it is very important to know how long each bug puts users at risk. Measuring exposure window encourages vendors to fix holes quickly, and to get those fixes out to users.

Complete Disclosure : The other measurements you compile are almost meaningless if you can’t see all the fixed bugs. Some vendors only disclose flaws found by outside sources, concealing those discovered by their internal security teams to keep their bug numbers down. Measuring only externally-discovered vulnerabilities rewards vendors who are purely reactive and, worse, it fails to credit vendors who develop strong internal security teams. Those teams often find the majority of security bugs; it’s important that any security metric recognizes and encourages it.

What’s The Solution?

If it were easy to find a calculation that included all of this information in a universal way, we’d be using it. When we wrote about our metrics project last year, it was with the aim to develop these ideas, and to change the tone of the discussion.

If the work there has taught us anything, it is that this will not happen overnight. The first step, though, is being clear about what we should expect from any assessment of security. If it doesn’t focus on the three SEC essentials: Severity, Exposure Window, and Complete Disclosure, ask yourself why not. And then ask the people doing the measuring.

Johnathan Nightingale
Human Shield

Category: Musings, Security | Tags: | 13 Comments »

CanSecWest 2009 Pwn2Own Exploit and XSL Transform Vulnerability

03.26.09 - 01:55pm

Issue

The pwn2own bug that Nils discovered at CanSecWest 2009 and the XSLT vulnerability recently made public by Guido Landi (http://www.securityfocus.com/bid/34235) are both critical issues that can result in malicious code execution.

Impact

These issues can be exploited by tricking a user into visiting a malicious web page hosting the exploit code. The pwn2own bug can be mitigated by disabling JavaScript.

Status

Both issues have been investigated and fixes have been developed which are now undergoing quality assurance testing. These fixes will be included in the upcoming Firefox 3.0.8 release, due to be released by April 1. You can follow our work in bugzilla.

Credit

The pwn2own bug was reported to Mozilla by Nils via the Zero Day Initiative (ZDI). The XSLT issue was discovered on http://www.milw0rm.com/exploits/8285, credited to Guido Landi.

Category: Firefox, Press, Security, Vulnerabilities | | 16 Comments »

New CSS Grammar Fuzzer

03.17.09 - 01:24pm

Mozilla’s Jesse Ruderman just blogged about a new CSS grammar fuzzer of his, to go along with the JS fuzzer we announced a while ago.

Fuzzers are a tool that we’ve found incredibly valuable in the past, and continue to employ heavily. A fuzzer’s job is to make your application fail by feeding it surprising inputs. The good ones do this by knowing a part of your code well enough that they can make smart guesses about how to confuse it. This one, for instance, produces a constant stream of mostly-correct CSS rules, and watches to see whether the browser can cope with them. Because fuzzers take these random paths, they can uncover subtle bugs that are rarely encountered during “normal” testing; and Jesse is a master at building them.

When Jesse originally started talking about his javascript fuzzer, he gave it to other browser vendors first, and he’s done the same with this one. If you’re interested in automated security analysis tools though, he’s now made it public, and I recommend checking it out.

Johnathan Nightingale
Human Shield

Category: Security | Tags: , | 2 Comments »

Beware the Security Metric

03.06.09 - 02:50pm

Security metrics are very difficult to do well, and easy to do poorly. For example, take a look at the recent Secunia “2008 Report” (http://secunia.com/gfx/Secunia2008Report.pdf). It tries to break down vulnerabilities reported by browser, and specifically states:

31 vulnerabilities were reported for Internet Explorer (IE 5.x, 6.x, and 7), including those
publicly disclosed prior to vendor patch as well as those included in Microsoft Security
Bulletins.

Safari and Opera each had 32 and 30 vulnerabilities, whereas 115 vulnerabilities were registered for Firefox in 2008.

From a quick read it appears as though Firefox had almost 4 times as many security issues as IE or Safari! Like, OMG! However, that conclusion would be painfully incorrect. Mozilla discloses and releases bulletins for all security issues fixed in Firefox, regardless of how they were discovered. Unlike other vendors that only disclose issues reported by external independent parties, but not by internal developers, QA or security contractors.

So presenting those numbers as comparable is worse than useless, it is in fact very misleading. It’s like comparing traffic accident rates for two cities of equal size, but one only reports accidents that make the news while the other reports all traffic accidents. Directly comparing such numbers is meaningless.

Some vendors make the point that the number of internally found issues is small and not meaningful. That would unfortunately imply their internal testing and security processes are incapable of finding security issues, and rely entirely on the generosity of random strangers (security researchers). I would find that pretty scary.

Fortunately, having worked in-house and consulted to a number of large software vendors, I can assure you that is not true. In fact they generally have very capable security teams and QA processes, which are so good at finding security issues that they usually find far more internally than they ever disclose to the public.

The Secunia report is deeply disappointing on a number of levels. Frankly, it’s disappointing that security researchers aren’t taking the “research” part of their jobs as seriously as they once did. It’s also disappointing that Secunia would publish something like this as one really expects better from them. This sort of reporting only encourages companies to hide as many security issues and fixes as possible, which moves the state of security backwards. And this is perhaps the most disappointing thing of all.

Lucas Adamski
Director of Security Engineering

Category: Firefox, Musings, Security, Uncategorized | | 29 Comments »

MD5 Weaknesses Could Lead to Certificate Forgery

12.30.08 - 08:41am

Issue

Researchers have recently found weaknesses in the MD5 hash algorithm, relied on by some SSL certificates. Using these weaknesses, an attacker could obtain fraudulent SSL certificates for websites they don’t legitimately control.

Impact to users

If a user visits an SSL site presenting a fraudulent certificate, there will be no obvious sign of a problem and the connection will appear to be secure. This could result in the user disclosing personal information to the site, believing it to be legitimate. We advise users to exercise caution when interacting with sites that require sensitive information, particularly when using public internet connections.

Status

This is not an attack on a Mozilla product, but we are nevertheless working with affected certificate authorities to ensure that their issuing processes are updated to prevent this threat. Mozilla is not aware of any instances of this attack occurring in the wild.

Microsoft has released their own advisory as well.

Credit

Alexander Sotirov, Marc Stevens, and Jacob Appelbaum presented this work at the 25th Chaos Communication Congress.

Johnathan Nightingale
Human Shield

Category: Security | | 19 Comments »

The Importance of Good Metrics

12.15.08 - 02:48pm

There has been some interest in the last few days about a recent report from a company called Bit9 about application vulnerabilities. While we’re always happy to see stories that focus on educating our users about security, there are some problems with Bit9’s methodology that hinder its ability to draw any meaningful conclusions.

Bit9 says it drew up this list by identifying popular applications that have had a critical vulnerability reported in 2008. This is an ineffective test, as it rewards software companies that conceal their security vulnerabilities. Mozilla focuses a great deal of energy on building world class code, and we stand by our reputation on security; we don’t play games with it.

Mozilla security process involves regularly identifying, fixing, testing, and releasing security updates to keep our users safe, and we do that in a public way so that others can scrutinize our processes and help make them better. To suggest that this openness is a weakness because it means that we have “reported vulnerabilities” is to miss the reality: that software has bugs. A product’s responsiveness to those bugs and its ability to contain them quickly and effectively is a much more meaningful metric than counting them.

Bit9 seems to understand this in its focus on application support for updates, but again it fails to account for the real world experience.  Firefox does not deliver WSUS updates, but our built-in update mechanism requires no user intervention, and we consistently see 90% adoption within six days of a new update being released.

The Firefox vulnerabilities Bit9 discusses are long-since fixed, with the majority of these fixes coming within days of it being announced.  That is the real measure of application security: are known vulnerabilities fixed promptly, tested carefully, and deployed thoroughly? When people have asked that question, Firefox and Mozilla have consistently come out ahead.

Bug counting is unfortunately common because it’s easy, but it should not be a substitute for real security measurement. This is why we’ve continued to work on things like the Mozilla security metrics project, to help people make informed decisions about the security of their software. We invite people who are interested to be a part of that process.

Johnathan Nightingale
Human Shield

Category: Firefox, Press, Security, Vulnerabilities | | Comments Off