Notes taken from
Boris Beizer books:
On what kind of bugs are we betting? The processing, whatever it is, must define a functional relation between the input variables. There are two possibilities for functional relations between input variables: (1) They appear in the same output function, or (2) they appear in the same domain boundary.
1. Same function. The calculated function is either right or wrong. We are assuming, by testing combinations, that it is not possible to see the bug except at a particular combination of input values (the extreme cases) because the program is correct at other combinations of input values. What we are saying, if reinterpreted, is that the normal processing and the erroneous processing for the combination of extreme values lie in different domains. If that's the case, we'd better check the domain boundaries.
2. Same domain boundary. The input variables might appear in the same (interpreted) domain boundary inequality. Either they do or they don't. If they don't, then they are independent. There is no functional relation between them and we are hypothesizing a bug that will create a relation where none existed. That's no bug, that's sabotage. If there is a functional relation, then that's what should be tested, because testing raw combinations of extreme points tells us nothing about such relations.
In the example of Figure 7.3 on the next page, I applied the heuristic combination strategy and got the points marked with hashes. We have three domains whose boundaries are given by the solid lines. Only eight of the test points-those adjacent to the upper boundary's extreme points-tell us anything at all about the correctness of the domains' boundaries. The tests tell us nothing at all about the correctness of the lower boundary. Suppose we had a bug in which the two domains were implemented with the boundaries shown by the dashed lines. Instead of three domains we have four domains, and who knows what kind of strange processing in them. The extreme point combination tests won't catch this bug.
"But," comes the objection from the sidelines, "we know that there are domains and we should have tested the inputs at and around X = 6 and Y = 6 and their combinations!" Better, but not good enough. This supposed refinement adds 12 tests but still doesn't catch the crossed boundary bug. It comes down to: (a) the only way to test a functional relation is to test the functional relation and (b) betting on a bug that creates functional relations where there were none is a bad bet.
Testing combinations of extreme points is a poor-man's n-dimensional domain testing. It generates many tests, most of which are meaningless and/or misleading. In the last example,we did 29 tests but didn't catch the bug. A proper strategy will find the bug with only four tests.The worst part about this "strategy" is that the number of test cases grows exponentially with the number of input variables. Consequently, even if automated, coverage is impossible. Only a small part of that awesome number of combinations (1.2677E30, for 50 inputs, or 4E19 years of testing at one test per millisecond) can be attempted so those that will be executed are effectively arbitrary. Properly designed random testing will find more bugs and tell you more about the software's dependability. Proper domain testing does even more for less work.
The "strategy" of using extreme point combinations, if it can be called that, assumes that there is an upper and lower acceptable value for every input variable. Below the minimum value and above the maximum value, presumably the software corrects or rejects such inputs. Obviously, the minimum and maximum values (in part) specify domain boundaries, and the strategy asks first that we test the domain boundary and then values just above the maximum and just below the minimum. That's for one dimension. For two dimensions, the strategy recommends that we test the combinations of extreme values. If I call the four interesting cases for each variable, respectively, "under," "min," "max," and "over," for two variables we would, under this strategy test: (underA, underB), (underA, minB), ... (overA, overB), for a total of 16 tests, plus one test for a typical value. For n input variables, the number of test cases needed to follow this prescription is 4n +1.
Functional relation between input variables.