SECTION A - General and Introductory

A1 - Generalized verbiage and introduction to the test plan documents

A2 - Glossary of terms used:

Most systems have their own vocabulary. Different organizations working on the same problem may use different terms to mean the same thing. The test plan documentation must be understood by designers, QA, and the buyer. A printed glossary helps avoid a lot of useless debate and misconceptions

A3 - Reference list to specifications, subsidiary specifications, design memoranda, correspondence, and so on:

The specification is not a complete document in itself. It may refer to other specifications and standards. Furthermore, there are pertinent correspondence and agreements, minutes of meetings, approved changes of scope, and so on that modify or contradict the original specification. The test plan documentation should include sufficient references to all such sources so that conflicting opinions over functions can be quickly resolved.

A4 - Test design standards and conventions:

The test plan is a formal document no less structured than code. If there are coding standards, there should be comparable test plan standards and conventions. These cover the structure and content of every test input or scenario. In communications system testing, as an example, we include rules for the construction of message addresses, sources, data and time, and every character of text. One of the standards makes every message unique, and where possible the message text itself identifies the test area, the purpose and intended outcome. Similar rules and conventions cover the construction of the data base, allocation and naming of terminals, allocation of resources, test scope, input method, verification methods, and so on.

A5 - Test running order, procedures, summary sheets, and control sheets:

While the test plan may be a linear document that starts with test group C1 and ends with Group C20, because different data bases and configurations are needed, it is not possible to guarantee that the entire plan can be executed in the documented order. Furthermore, discovered flaws, the need for retesting, or schedule or personnel problems may force changes in he running order. Test control documents are need to specify the running order of the test, and to monitor progress and status of what could be a few thousand sub tests.

SECTION B - Test Data Base and Code

B1 - General

B2 - Test data bases documentation and cross reference. One subsection for each variant data base:

The number of test databases should be minimized because each variant data base must be tested and debugged. Variant data bases are used when 1) there is no other way to create the condition needed by the test, 2) when test execution time would be excessive under the normal data base, 3) when test design can be simplified or made more effective by using a special data base, 4) when inadequate hardware resources (e.g. memory) does not permit testing all features at their extreme points under a single data base.

B3 - Support programs or code documentation for test data generators, drivers, and load generators:

Special programs may be written for generating transactions or load. Alternatively, a pre-existing load generator with parameters set for the test may be employed. All such code or specifications must be documented either directly or by reference as part of the test plan documentation.

B4 - Test configuration specification. Specification of hardware and software configurations needed for different tests:

The test bed may be a superset of actual installations for multi-site systems or may be a subset of an ultimate configuration which is not fully implemented in the test bed. Either of these situations means hat there is more than one hardware configuration to deal with. Similarly, there may be several different operating systems under which an application system is to run. Each such hardware/software test configuration must be defined and documented in detail.

B5 - Test tools hardware and software:

Documentation of all specialized test tool software and hardware not covered elsewhere. Directly if unique to this test plan, indirectly by reference with an appropriate specification of controlling parameters if pre-existing hardware and/or software is used.

B6 - Verification hardware and software:

Some aspects of testing, such as performance testing, may require specialized hardware and/or software for verification. Specially installed instrumentation software or hardware used to confirm transaction flow paths, or off-line historical or log analyzers may be used. All such hardware and software must be documented directly or by reference.

B7 - Miscellaneous support hardware and software used to generate and execute the test:

A catch-all for anything not covered in the above categories. For example, program patches or modifications that are used to abet testing if they are used in more than one test area.

SECTION C - The Actual Test Specifications

Structure and scope:

GROUP:

the broadest functional subdivision of the system. A typical system will have ten to twenty test groups. A typical test group might be the hardware configuration group, which tests all the variant hardware configurations.

SUBGROUP:

A functional subdivision of the group which can be designed and executed independently of any other subgroup. For example, a configuration test that shows that any disk drive can serve in any functional capacity would be covered under the rotation test subgroup.

TEST:

A test is a subdivision of a subgroup that contains one or more sub tests. Every sub test is a test, in addition to covering closely related functional areas, shares the same setup, terminals, verification method, and data base with every other sub test in that test. In principle, once a test has been set up, its component sub tests can be run through mechanically.

SUBTEST:

The smallest division of the test plan. A relatively microscopic subdivision that includes a detailed specification of inputs and outcomes. In principle, each sub test has exactly one input and one outcome. A sub tests is indivisible. When a sub tests consists of several steps, the entire sub tests must be repeated from the beginning for it to make sense. If the system fails any part of a sub tests, the entire sub tests must be repeated.

Sample Groups:

STARTUP and INITIALIZATION

OPERATOR COMMANDS

CONFIGURATION CONTROL

INTERFACES AND PROTOCOLS

RESTART

SWITCHOVER AND RECOVERY

PERFORMANCE TESTING

STRESS TESTING

TERMINAL MANAGEMENT

OFF-LINE DATABASE GENERATOR

ON-LINE TABLE CHANGES

SECURITY TESTING

---

Sample Test Sheet:


GROUP _______ OF ________
TEST _______ OF ________
SUBTEST_______ OF ________
PAGE _______ OF ________


SPECIFICATION REF:

VALID ACTION TEST _______
INVALID ACTION TEST _______

TEST DESCRIPTION



SUBTEST DESCRIPTION


SUBTEST OBJECTIVES


SETUP







INPUT SPECIFICATION, MESSAGE, OR REFERENCE(S)






EXPECTED OUTPUT(S), SPECIFICATION, MESSAGE, OR REFERENCE(S)





RELEASE / / FIRST RUN / / FORMAL RUN / / STATUS PASS/ FAIL
RETEST / /