Timr Testing Guy
Testing is like a good cake… Layered
Getting going with project metrics
July 28th, 2009 by timr
Murali and I have been working on project metrics trying to figure out what data we have, how it can be used, and how to automate the collections, analysis, and reporting. Some people know about our activities. We did a brown bag and a few other communications, but nowhere near enough. Sorry about that. So we are going to amp this up a bunch.
What do I mean by project metrics? I am talking taking data from Bugzilla, code coverage test runs, and Hg.
- Bugzilla data includes: fix rates, find rates, blocking bugs, unconfirmed bugs, time to confirm bugs, time to find bugs by the community, which components have the most regressions, security bugs, or crashes, etc.
- Coverage data comes from instrumented builds. We run the tests (both automated and manual!) against these builds and collect code coverage data.
- Finally, we look at the Hg check-ins to see which files change the most and are changed by which bugs.
From all this, for example, we can determine which code from Hg has the most regression bugs and the lowest code coverage from our tests. We end up with some pretty interesting mash-ups!
We are not doing this in isolation. We are working with Simon from metrics team. Simon is working on exposing Bugzilla metrics via Pentaho tools. Murali and I are using Buzilla data as one step to creating interesting mash-ups with other data sources. We are working with Simon to use Pentaho to organize and present the data in scalable and flexible ways. However, currently Pentaho data cannot be made public. So in the interim, we are producing some scripts that do the analysis on aggregate tables from Bugzilla and pull data from HG and the code coverage results. We have a working prototype of these scripts producing charts with a variety of views.
Some stuff we have figured out:
- What is available in the open source and commercial code coverage tools. Evaluated several code coverage tools including Bulls eye and PureCoverage. ctalbert was a big help with this!
- How to instrumented builds, collect data, and process code coverage data from C/C++ code using gcov
- How to collect and process JS code coverage (thanks Ed Kelley from JSCoverage for making a fix for us!)
- How to more reliably get coverage results given that any one build may be red or orange (pending a integration by catlee)
- How to create aggregate tables of Bugzilla data to minimize the impact to the live server and to speed dynamic queries/drill downs.
- How to correlate bugs with certain keywords with specific files, number of Hg checkins to those files along with coverage stats to come up with potentially highest risk files.
So what is next?
- Finalize the initial blocking related metrics and provide them for the project meeting within a week
- Provide vital stats for every source files: total fixed bugs, # of regression bugs, # of security bugs, # of crasher bugs, manual BFT coverage (% of lines cov), automated unit test coverage (% of lines cov), and function coverage by file (automated tests and manual). From this we can provide the intersection of functional coverage between automated and manual tests for a given source file (overlap = potential areas to prune manual tests). We can also provide exclusive functional coverage for automated and manual tests. These are areas where manual tests are certainly valuable, but might also be areas that we should convert manual tests to automated tests.
- Identify files of interest: Files that have high regression rates, low code coverage, and/or lots of check ins. These should be easy targets for community test development projects!
- Investigate how we can get more community attention on unconfirmed bugs and the possible major issues hiding there
One of the goal it to provide interesting and useful visualizations. We have done some tests with bubble diagrams, interactive charts, etc. As a final thought, it would be great to find he project metrics of this this cool data visualization. It is compact while also content-full and look great!
This just rocks. I look forward to using the fruits of this labor in our accessibility code.
This project is pretty advanced for the time. Sort of like a back to the future car in the 80’s
But it is very cool and appears to be working. Being in an open source environment, it would be nice if this could be packaged up as a toolkit for others to contribute to as well.
I know there is a lot of talk about reducing the overall unittest runtime, maybe finding the coverage for each test, then using that data to find redundant tests or pieces of tests which could be removed if logically it looks like duplicated functionality.
One other piece of data that would be good to know is for all the manual tests that are run per release (say smoketests), can we ensure we have enough logical functionality and measured code coverage to hit 90%+ with a half hour of manual testing? Finding ways to harvest that data would be very beneficial.
The only danger I see is when using this data as more than a spotcheck of where we are (vs using it as checkin criteria, or worse yet hiring/firing decisions), is that people will game the system. I saw this first hand in the past where code coverage was required before checkin and when deadlines are tight (think zero day fix) it is tempting to cheat especially if you are burned out.
I have seen a lot of work put into this project and would really like to see this available to other projects (if they are using bugzilla and hg or are fine adding support for their own bug/sourcecontrol).
Follow-up note: We have code coverage runs from July25 here:
http://people.mozilla.com/~mnandigama/codecoverage_html/
Any plans to update https://wiki.mozilla.org/QA:CodeCoverage ?
Yeah, we’ll update that.
Hi Tim.
You might want to check this http://weblogs.java.net/blog/cos/archive/programming/index.html – it might complement a couple of ideas.
Take care,
Cos
Very cool to see this happening. I recently set up a Metrics page on the wiki to link to all of the publicly available community data I was aware of. I’m sure you know of lots more. Please feel free to update this page or let me know if something like this already exists.
https://wiki.mozilla.org/Metrics
Thanks for pointing this out. I added some links under the Firefox section for the c/c++ and JS coverage numbers plus links to presos we have done. Also added some links to tools in the bottom references section.
I am fascinated by static analysis also. Thanks for the pointer!