The metrics team wants to reach out to the community of Bugzilla users to learn what would benefit them by asking: In Bugzilla, what kind of analytic questions should be answered? We have some analytic questions that we can answer, and we have gathered some questions that people would like to have answered. This blog post will outline these questions so you can contribute new questions to help us analyze the correct places. What would you like to see analyzed in Bugzilla?
What we currently have
The current Bugzilla SQR(Software Quality Reports) is placed in a star schema. A star schema is based on measures and dimensions. Measures are the main component being analyzed and can be aggregated. A good example of a measure would be sales dollar from a order. Dimensions are the “by” conditions. A good example of a dimension would be by time, or by product. Putting measures and dimensions together we can create sales dollars by day, sales dollars by week, and sales dollars by month or sales dollars by product. We can further group measures and dimensions in to cubes. Cubes are the collection of specific measures and dimensions. Right now the SQR has two cubes, Bugs, and Bugs Changes. With the SQR cubes we can ask things like: Bug Burndown By Product, Current Issues by Status and Product, Average Days to Resolution by product and priority, Overall open vs. closed trend for bugs by product. Below are all the details of each cube.
Bugs Cube
Measures:
- Bugs
- Days Since Creation
Dimensions:
- Time Created
- Bug Type
- Person Assignee
- Person Reporter
- Priority
- Product
- Status
- Time Release
- Version
Bug Changes Cube
Measures:
- Changes
- Actual Issues
- Opened
- Resolved
- Net Open
- Starting Net Open
- Ending Net Open
- Days Since Creation
- Elapsed Days
Dimensions
- Time Created
- Issue Type
- Person Assignee
- Person Reporter
- Priority
- Product
- Status
- Time Released
- Version
Current Analytic Questions Raised
We have talked to a few members of MoCo already and have also seen a few projects as well. Justin Scott has came up with an interesting project to analyze Bugzilla. Damon Sicore has a project that looks at Blockers By Priority, Blockers Closed By Day, Non-Blockers Closed by Day, Blockers added by Day, and a Break down of bugs being closed by person in each team. Murali Nandigama is a member of the QA team at Mozilla corporation and has raised a few Bugzilla metric questions of his own already. Murali is interested in: What are the top components where there are a lot of incoming critical and blocker bugs and how fast are we able to triage them to confirm or not?, After the bug is confirmed, how long are we taking to fix it?, After the bug is fixed how long are we taking to verify?, What is the mean time between confirmed to verify of bugs by components?, What is the re-fix rate and which component have significant re-fix rates?, and What is the duplicate bugs rate and which components have significant duplicate rates?
Now the metrics team wants your input. What would you like to see?
I’m interested in this information, too! :-) The Bugzilla Project would be very interested in finding out what you come up with as results, here.
-Max
Bugzilla is a very interesting beast, it’ll be very cool to see what sort of questions do get asked as well as what the answers are.
There’s always a lot of debate about the UI, this sounds like it’ll really help define the bugzilla workflow and make it easier to improve it.
I think something important to measure would be time from a community submitted patch to some sort of response or review. Or even what areas are we getting outside interest in.
(Component) Reliability would be an interesting metric.
Counting the number of bugzillas (where STATUS=’NEW’) for a given component, one can use a time factor to determine the number days elapsed in between to determine reliability.
The lower the number of days, the lower the reliability. The higher the number of days, the component is judged more reliable…
I’d like to see tracking of bugs that get closed and then re-opened at a later date. Not re-fixes (failed to pass QA after being marked resolved), but regression on fixed bugs (passed QA, but someone used old code in a later build or broke it while fixing another bug).
@Bob
Could you tell us a little about how we might be able to detect the difference between a re-fix, a regression, and any other sort of re-opening?
@deinspanjer – I view them as follows:
Re-fix would be a bug moving from Resolved-Fixed to Reopened
(failed the QA verification)
Regression would be a bug moving from Closed to Reopened
(passed QA but was reopened later)
This could mean several things that the organization would need to identify internally.
QA improperly passed the fix when it should have been rejected.
Maybe it was fixed, verified, closed, and then got screwed up again later.
It simply may not be possible with Bugzilla as it currently exists.
How quickly bugs go from being filed to the VERIFIED state. Then a breakdown as follows, to identify which areas take the longest:
How quickly bugs go from UNCONFIRMED to NEW, NEW to ASSIGNED, ASSIGNED to RESOLVED and RESOLVED to VERIFIED. Perhaps on a per product basis as well as overall. This is the best way to assess the general workflow I think.
The “velocity” metric Dan proposes seem to overlook bugs that never make it to production.
If a developer fixes a bug with the wrong branch of code, a QA person will quickly catch the regression and re-open the bug.
Consequently, the velocity of this bug would be extremely low and fit outside of the “data cloud”.
My 2 cents.
[...] – there’s a current project underway aimed at answering some key analytical questions related to Bugzilla and providing folks with a [...]