Timr Testing Guy
Testing is like a good cake… Layered
Getting going with project metrics
July 28th, 2009 by timr
Murali and I have been working on project metrics trying to figure out what data we have, how it can be used, and how to automate the collections, analysis, and reporting. Some people know about our activities. We did a brown bag and a few other communications, but nowhere near enough. Sorry about that. So we are going to amp this up a bunch.
What do I mean by project metrics? I am talking taking data from Bugzilla, code coverage test runs, and Hg.
- Bugzilla data includes: fix rates, find rates, blocking bugs, unconfirmed bugs, time to confirm bugs, time to find bugs by the community, which components have the most regressions, security bugs, or crashes, etc.
- Coverage data comes from instrumented builds. We run the tests (both automated and manual!) against these builds and collect code coverage data.
- Finally, we look at the Hg check-ins to see which files change the most and are changed by which bugs.
From all this, for example, we can determine which code from Hg has the most regression bugs and the lowest code coverage from our tests. We end up with some pretty interesting mash-ups!
We are not doing this in isolation. We are working with Simon from metrics team. Simon is working on exposing Bugzilla metrics via Pentaho tools. Murali and I are using Buzilla data as one step to creating interesting mash-ups with other data sources. We are working with Simon to use Pentaho to organize and present the data in scalable and flexible ways. However, currently Pentaho data cannot be made public. So in the interim, we are producing some scripts that do the analysis on aggregate tables from Bugzilla and pull data from HG and the code coverage results. We have a working prototype of these scripts producing charts with a variety of views.
Some stuff we have figured out:
- What is available in the open source and commercial code coverage tools. Evaluated several code coverage tools including Bulls eye and PureCoverage. ctalbert was a big help with this!
- How to instrumented builds, collect data, and process code coverage data from C/C++ code using gcov
- How to collect and process JS code coverage (thanks Ed Kelley from JSCoverage for making a fix for us!)
- How to more reliably get coverage results given that any one build may be red or orange (pending a integration by catlee)
- How to create aggregate tables of Bugzilla data to minimize the impact to the live server and to speed dynamic queries/drill downs.
- How to correlate bugs with certain keywords with specific files, number of Hg checkins to those files along with coverage stats to come up with potentially highest risk files.
So what is next?
- Finalize the initial blocking related metrics and provide them for the project meeting within a week
- Provide vital stats for every source files: total fixed bugs, # of regression bugs, # of security bugs, # of crasher bugs, manual BFT coverage (% of lines cov), automated unit test coverage (% of lines cov), and function coverage by file (automated tests and manual). From this we can provide the intersection of functional coverage between automated and manual tests for a given source file (overlap = potential areas to prune manual tests). We can also provide exclusive functional coverage for automated and manual tests. These are areas where manual tests are certainly valuable, but might also be areas that we should convert manual tests to automated tests.
- Identify files of interest: Files that have high regression rates, low code coverage, and/or lots of check ins. These should be easy targets for community test development projects!
- Investigate how we can get more community attention on unconfirmed bugs and the possible major issues hiding there
One of the goal it to provide interesting and useful visualizations. We have done some tests with bubble diagrams, interactive charts, etc. As a final thought, it would be great to find he project metrics of this this cool data visualization. It is compact while also content-full and look great!
Plans for Q3 2009
July 22nd, 2009 by timr
Here are some updates about our plans for this quarter (July-Sept 2009). We’ll be testing Firefox security releases along with the early releases for Firefox 3.6. Mobile is really heating up with early releases on Nokia devices, Windows Mobile devices, and SmartBooks.
The next area is making existing test tools and frameworks easier to use by the community. We have created a bunch of tools and leveraged existing tools. Now it is time to mkae them more user friendly and accessible to more members of the community. Some key areas here are more automated JS tests, QAC improvements, automated crash reproduction tools, and Litmus.next.
Next up is continuing our drive to automate manual tests. The manual Firefox smoke tests are now automated (Woo Hoo!!). It used to take 20 minutes platform to run these tests manually. The MozMill based automated smoke tests take only 2 minutes to run!!
The final focus for this quarter is developing community involvement. Here we are holding more meet-ups with potential and current community members. We have had great success with themes based meetups. Like crash reporting and analysis and testing web properties. We have also held meet-ups in Washington DC and Berlin Germany. And generally we want to do whatever we can to attract passionate new community members and continue to engage existing members.
Here is the detail list of things we are planning this quarter:
- Directly contribute to the successful releases of Firefox, Fennec, and prepare for 3.x testing
- Test Firefox 3.5.x, 3.0.x,
- Test Fennec milestones
- Test Major web sites (AMO, SUMO, mozilla.com, etc)
- Integrate WinMobile unit tests in build testing (tunit in particular)
- Create test tools and frameworks for capabilities of new features in 3.next: HTML5, HW Acceleration Gfx, Multi-process
- create manual BFTs for AMO, SUMO, and Spreadfirefox to ensure consistent regression testing.
- Make existing test tools and frameworks easier to use by community. Examples:
- Create automatic crash reproduction by stitching together existing tools from a manual process.
- Complete remaining work to fully automate the JS reftests – provides automated testing along with improved ability for JS devs to run and write tests
- Hook Fennec test results into new results server
- Fix top issues with QAC
- Port QAC for Fennec – make it easier for community to file bugs!
- Work with WebDev to fix most of the QMO P1 bugs (currently 14 P1s)
- Major improvements to Litmus – resolve performance issues, improve ability for web testers and l10n testers to use tool
- Make the coverage metrics more visible and reliable along with evangelize their use in the test development process
- Continue drive to automate manual tests
- Focus for this quarter will be the BFTs (Basic Functional Tests) and we will consider filling coverage gaps as we go along. We will target components based on the most critical features and lowest coverage. Specific components TBD.
- Integrate these tests into the buildbot testing framework. Joint goal with RelEng for review and landing!
- Automate key areas of the web testing BFTs with Selenium, for example. Key areas TBD.
- Develop Community involvement. Examples:
- Hold regular monthly meet-ups in Mountain View
- Attract new community members: Find 2-3 new community members to help with test development, test tools development, and/or multiple test execution sessions (ex: bug triage, fix verification, release testing)
QA plans this Quarter
July 11th, 2008 by timr
I need to get better about communicating so here goes.
The QA team has been getting better and better about planning goals and thinking more strategically. This quarter we still have a lot of release testing to do (a lot = understatement!). Here is the current list of planned releases for this quarter:
- FFx 3.0.1
- FFx 3.0.2
- FFx 2.0.0.15
- FFX 2.0.0.16
- FFX 2.0.0.17
- TBird 2.0.0.16
- TBird 2.0.0.17(?)
- FFx Major Update 2.0.0.x -> 3.0.x
- Partner releases for 3.0.1
- FFx 3.1 Alpha
- FFx 3.1 Beta 1 (?)
- SpiderMonkey (JavaScript) 1.7
- SpiderMonkey 1.8
- QMO Beta
Beyond release testing, we really want to get better at automated testing. Some great efforts here are the develpoment of Gristmill, a lightweight test scripting tool for the Firefox GUI. Once this is done, then the test execution team can automate many test cases in the Litmus web based manual testing tool.
We are also planning a series of brown bag presentations on Gristmill, Mochitest, Reftest, Selenium, and creating simplified test cases for bugs with javascript. These are mainly to raise the understanding of these tools and techniques within the QA team, but also for anyone else that is interested. Hopefully we can record these and publish them on devmo.
Finally, the test execution team will be creating test cases with Gristmill, Mochitest, and Selenium to get more familiar with these tools and start automating our manual tests.
That’s just scratching the surface for what QA is up to for this quarter. I’ll post more soon.
Hello world!
February 19th, 2008 by timr
Welcome to Blog.mozilla.com. This is your first post. Edit or delete it, then start blogging!