Solutions To Imagined Problems
March 5th, 2007
The Mozilla project has been ramping up its unit testing efforts. Everyone agrees this is a good thing, but there is no consensus on who should be writing which tests. Here’s my two cents.
The typical argument is one along the lines of “Genius Q. Corehacker should have someone writing tests.” This idea is worth considering in some situations, but it underestimates the costs of such an arrangement in several ways. The first is that the relationship must be close to symbiotic, otherwise communication overhead makes the arrangement unworthwhile. My feeling is that we should watch for these in Bugzilla, but the corporation should not try to create them through hiring. The second problem is one of prioritization. The surgical team is an appealing construct. However, I think it is necessary to classify basic test cases as the responsibility of the domain expert. Later, these basic test cases can be expanded by others, but a limited operational definition of correctness must be recorded by those who know the code. Otherwise, we risk a negative feedback loop, where testing questions from others consume a growing percentage of an engineer’s time.
The upshot is that new code doesn’t need to come with exhaustive test cases, but it should come with something. In otherwords, don’t try to write test cases for bugs you predict will happen in the future. Write them for bugs you’ve hit yourself.
Generating coverage retroactively is a harder problem. Authoring tests is an excellent introduction to the hacking process. We can probably get decent sanity checks in every module by having people do that, but we’ll never cover ten years of bugs that way. We should look to automated solutions from test engineers, so we can cover bugs without increasing the channels of communication.
Last but not least, having engineers writing at least some tests with checkins is known to work. Are there any open source projects using [YOUR PROCESS SUGGESTION HERE] that are considered well-tested?
P.S. – Unit tests are not a spectator sport. If you have written zero unit tests, you know know nothing about how to write them, how much time they take to write, how much time they save, their effect on code quality, or the degree to which testing knowledge for a particular module is transferable. The first test is painful to write. I don’t know of many people who’ve stopped writing them after the tenth or so.
March 6th, 2007 at 9:40 am
As I understand them (please correct me if I’m wrong), unit tests are generally more complex than individual bug test cases. Is this why you say unit tests are so much harder?
March 6th, 2007 at 10:55 am
Unit Testing is kind of an all or nothing paradigm. New code should be checked in full unit tested so that you have at close to 100% code coverage in that class/andor function. Ideally you’d also have tests for the cyclomatic complexity which is the level of paths code can travel in your script. IE how many if/elses are there and are you testing for all conditions.
When a bug comes in you simply add a unit test for that particular bug and move on. Unit testing has saved my but many a time and I’ll never go back to “checkin.. please don’t break anything” lol
March 9th, 2007 at 3:49 pm
I think the definition of a unit test is pretty simple. A unit test tests each method or function in a class’s interface for basic conformance. Does each method basically do what it is supposed to do?
Coverage is much more complicated than this. Unit tests may get one near to method coverage, but not anywhere near branch coverage.
Tests that come out of bugs are often not unit tests. They are more often a variance test. A variance test will test how something responds when things go wrong, or under anomalous conditions. Think “edge cases”. Just getting tests from bugs will not get you unit testing.