Comments on: Getting going with project metrics http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/ Testing is like a good cake... Layered Thu, 30 Jul 2009 21:05:48 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: timr http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-13 timr Thu, 30 Jul 2009 21:05:48 +0000 http://blog.mozilla.com/tim/?p=13#comment-13 I am fascinated by static analysis also. Thanks for the pointer! I am fascinated by static analysis also. Thanks for the pointer!

]]> By: timr http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-12 timr Thu, 30 Jul 2009 21:05:00 +0000 http://blog.mozilla.com/tim/?p=13#comment-12 Thanks for pointing this out. I added some links under the Firefox section for the c/c++ and JS coverage numbers plus links to presos we have done. Also added some links to tools in the bottom references section. Thanks for pointing this out. I added some links under the Firefox section for the c/c++ and JS coverage numbers plus links to presos we have done. Also added some links to tools in the bottom references section.

]]> By: David Boswell http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-11 David Boswell Thu, 30 Jul 2009 19:04:15 +0000 http://blog.mozilla.com/tim/?p=13#comment-11 Very cool to see this happening. I recently set up a Metrics page on the wiki to link to all of the publicly available community data I was aware of. I'm sure you know of lots more. Please feel free to update this page or let me know if something like this already exists. https://wiki.mozilla.org/Metrics Very cool to see this happening. I recently set up a Metrics page on the wiki to link to all of the publicly available community data I was aware of. I’m sure you know of lots more. Please feel free to update this page or let me know if something like this already exists.

https://wiki.mozilla.org/Metrics

]]> By: Konstantin Boudnik http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-10 Konstantin Boudnik Wed, 29 Jul 2009 20:51:56 +0000 http://blog.mozilla.com/tim/?p=13#comment-10 Hi Tim. You might want to check this http://weblogs.java.net/blog/cos/archive/programming/index.html - it might complement a couple of ideas. Take care, Cos Hi Tim.

You might want to check this http://weblogs.java.net/blog/cos/archive/programming/index.html – it might complement a couple of ideas.

Take care,
Cos

]]> By: timr http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-9 timr Wed, 29 Jul 2009 16:20:32 +0000 http://blog.mozilla.com/tim/?p=13#comment-9 Yeah, we'll update that. Yeah, we’ll update that.

]]> By: Ludovic http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-8 Ludovic Wed, 29 Jul 2009 14:54:23 +0000 http://blog.mozilla.com/tim/?p=13#comment-8 Any plans to update https://wiki.mozilla.org/QA:CodeCoverage ? Any plans to update https://wiki.mozilla.org/QA:CodeCoverage ?

]]> By: timr http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-7 timr Wed, 29 Jul 2009 06:03:38 +0000 http://blog.mozilla.com/tim/?p=13#comment-7 Follow-up note: We have code coverage runs from July25 here: http://people.mozilla.com/~mnandigama/codecoverage_html/ Follow-up note: We have code coverage runs from July25 here:

http://people.mozilla.com/~mnandigama/codecoverage_html/

]]> By: Joel Maher http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-6 Joel Maher Wed, 29 Jul 2009 02:08:28 +0000 http://blog.mozilla.com/tim/?p=13#comment-6 This project is pretty advanced for the time. Sort of like a back to the future car in the 80's :) But it is very cool and appears to be working. Being in an open source environment, it would be nice if this could be packaged up as a toolkit for others to contribute to as well. I know there is a lot of talk about reducing the overall unittest runtime, maybe finding the coverage for each test, then using that data to find redundant tests or pieces of tests which could be removed if logically it looks like duplicated functionality. One other piece of data that would be good to know is for all the manual tests that are run per release (say smoketests), can we ensure we have enough logical functionality and measured code coverage to hit 90%+ with a half hour of manual testing? Finding ways to harvest that data would be very beneficial. The only danger I see is when using this data as more than a spotcheck of where we are (vs using it as checkin criteria, or worse yet hiring/firing decisions), is that people will game the system. I saw this first hand in the past where code coverage was required before checkin and when deadlines are tight (think zero day fix) it is tempting to cheat especially if you are burned out. I have seen a lot of work put into this project and would really like to see this available to other projects (if they are using bugzilla and hg or are fine adding support for their own bug/sourcecontrol). This project is pretty advanced for the time. Sort of like a back to the future car in the 80′s :) But it is very cool and appears to be working. Being in an open source environment, it would be nice if this could be packaged up as a toolkit for others to contribute to as well.

I know there is a lot of talk about reducing the overall unittest runtime, maybe finding the coverage for each test, then using that data to find redundant tests or pieces of tests which could be removed if logically it looks like duplicated functionality.

One other piece of data that would be good to know is for all the manual tests that are run per release (say smoketests), can we ensure we have enough logical functionality and measured code coverage to hit 90%+ with a half hour of manual testing? Finding ways to harvest that data would be very beneficial.

The only danger I see is when using this data as more than a spotcheck of where we are (vs using it as checkin criteria, or worse yet hiring/firing decisions), is that people will game the system. I saw this first hand in the past where code coverage was required before checkin and when deadlines are tight (think zero day fix) it is tempting to cheat especially if you are burned out.

I have seen a lot of work put into this project and would really like to see this available to other projects (if they are using bugzilla and hg or are fine adding support for their own bug/sourcecontrol).

]]> By: David Bolter http://blog.mozilla.com/tim/2009/07/28/getting-going-with-project-metrics/comment-page-1/#comment-5 David Bolter Wed, 29 Jul 2009 01:53:19 +0000 http://blog.mozilla.com/tim/?p=13#comment-5 This just rocks. I look forward to using the fruits of this labor in our accessibility code. This just rocks. I look forward to using the fruits of this labor in our accessibility code.

]]>