Test Model Overview

The testing process starts with a program embedded in an environment, such as a computer, an operating system or it is in an embedded device. Any testing model has an implicit recognition and understanding of human suceptibility to error. This understanding leads us to create three models: a model of the environment in which the program is situated; a model of the program itself; and a model of the bugs expected.

From these models sets of tests are created and executed, giving either an expected or an unexpected result. An unexpected result may lead us to revise any or all of: the test; the model or concept of how the program behaves; our concept of what bugs are possible; or the program itself. Only in the rarest exceptions would the model of the environment or the environment (i.e., terminals, operators, LAN connections, operating platforms such as .NET, etc.) be subject to question given an unexpected result.

The Environment

Hardware and firmware are stable so it is not necessary to consider all of its complexity. Instead we work with a simplified model of the program's environment that includes our beliefs about the language, the operating system, the instruction sets and so on. An unexpected test result might lead us to revise our model of the environment, but it is very rare that the test result is indicating a bug in the environment itself.

The Program

Most program are too complicated to be understood in detail. We must simplify our concepts of the program in order to test it. Therefore, while a real program is being exercised on the test bed, we are in our brains dealing with a simplified version of it: a version in which most of the details are ignores. For example, if the program calls a subroutine, we tend not to think about the subroutine's details unless its operation is suspect. If the simplified model of the program does not explain an unexpected behavior we may have to modify that model to include more facts and more detail. And if that fails, we may have to modify the program (i.e. debug the actual code and fix it.)

Bugs

Our notion of what a bug is can vary. A poor specification may lead to proper behavior being taken as a bug and vice versa. An unexpected test result may lead us to change our notion of what constitutes a bug in a given context, and that means adjusting our model of bugs.

Tests

Tests are formal procedures. Inputs must be prepared, outputs predicted, tests documented, commands executed, and result observed; all of these steps are subject to error. There is nothing magical about testing and test design that immunizes testers against bugs. An unexpected test result is as often due to a test bug as it is due to a real bug. Bugs can creep into the documentation, the inputs, and the commands and cloud our observations of the results. An unexpected test result, therefore, may lead us to revise the tests. Because the tests are themselves in an environment, we also have a mental model of the tests, and instead of revising the tests, we may have to revise that mental model.

The Role of Models

Testing is a continuous process in which we create mental models of the environment, the program, human nature, and the tests themselves. Each model is used until either correct behavior is demonstrated or until the model is no longer sufficient for the purpose. unexpected test results always force a revision of some mental model, which in turn may lead to a revision of that which is being modeled. The revised model may be more detailed, which is to say more complicated, or more abstract, which is to say simpler. The art of testing consists of creating, selecting, exploring, and revising models. Our ability to do this depends on the number of different models we have at hand and their ability to express a program's behavior.