Skip to content

Instantly share code, notes, and snippets.

@textarcana
Created October 19, 2011 14:14
Show Gist options
  • Save textarcana/1298405 to your computer and use it in GitHub Desktop.
Save textarcana/1298405 to your computer and use it in GitHub Desktop.
Boris Beizer on why test suites wear out. Excerpted from Software Testing Techniques, 2d. Ed 1990

The following is excerpted from Software Testing Techniques, 2d. Ed. by Boris Beizer.

First Law: The Pesticide Paradox -- Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.

That's no too bad, you say, because at least the software gets better and better. Not quite!

Second Law: The Complexity Barrier -- Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.

...

Corollary to the First Law -- Test suites wear out.

Yesterday's elegant, revealing, effective test suite will wear out because programmers and designers, given feedback on their bugs, do modify their programming habits and style in an attempt to reduce the incidence of bugs they know about. Furthermore, the better the feedback, the better the QA, the more responsive the programmers are, the faster those suites wear out. Yes, the software is getting better, but that only allows you to approach closer to, or to leap over, the previous complexity barrier. True, bug statistics tell you nothing about the coming release, only the bugs of the previous release -- but that's better than basing your test technique strategy on general industry statistics or myths. If you don't gather bug statistics, organized into some rational taxonomy, you don't know how effective your testing has been, and worse, you don't know how worn out your test suite is. The consequences of that ignorance is a brutal shock. How many horror stories do you want to hear about the sophisticated outfit that tested long, hard, and diligently -- sent release 3.4 to the field, confident that it was the best tested product they had ever shipped -- only to have it bomb more miserably than any prior release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment