Testing and Documentation Examples
Testing Strategies
The following tips discuss how to approach testing.
There are two main types of problems presented in a program:
Note, some situations are a combination of the two, so the testing techniques have to be combined.
These examples illustrate some approaches to handling these problems.
General Comments
A test document demonstrates aspects of a program's correctness to the marker.
- Organize tests into categories to make it easier for the marker to find and understand test cases.
- Layout the purpose of each test case, the choice of data (if applicable), the expected results, and the aspects of the program being exercised.
- Show the results and discuss if the results are correct (i.e., passed the test) or incorrect.
If a test fails, say so rather leaving the case out as part marks are given for developing the test cases.
- Use line numbers to connect output to test cases in explanations.
Line numbers can be generated using shell command nl (see man nl).
- Note, you must test your program, not a given sample solution.
Test cases for a non-running program, receive no marks.
Data Problems
Invalid data comes from the command-line and/or files (including standard input).
- Organize data tests into categories, such as valid and then invalid input, and within these categories different valid results and errors.
To keep testing tractable, only test a representative set of values within each category.
For example, if the valid input consists of the letters a to z, there is usually no need to test all values in that range.
Similarly, there is no point in testing all invalid digits and punctuation characters, unless they have different behaviours.
- Always test border/boundary cases.
For example, given a range of data values, test some values below/above, start/end, and within the range.
- If data comes from a file and may contain invalid data, test cases involve a non-existent file, a file that cannot be opened for reading, an empty file, and file containing valid/invalid data.
(Whether or not an empty file is valid or not depends upon the problem specification.)
Similar kinds of test cases must exist for output files.
- If a program is supposed to treat standard input and an input file in the same way, demonstrate all tests for one of the input methods, and then show the results produced by the other method match using a shell command like cmp or diff.
- If the program specification uses the phrase assume error-free input or assume syntactically-correct input, these phrases do not mean the same thing.
In the first case, testing does NOT involve invalid data, and so the testing focus is only on valid data-cases.
For example, if the input is supposed to be a positive number, all data values are positive numbers.
In the second case, while the input has valid format, the actual values may be invalid.
For example, if the input is supposed to be a positive number, all data values are at most numbers.
The program must deal with invalid values like 0 or negative numbers.
Control-Flow Problems
Invalid control flow occurs when the program transfers to the wrong location.
- The logical-expressions of a conditional or looping statement must be tested to ensure correct control-flow.
- The different forms of call/return also significantly affect control flow and must be tested.
- As well, communication of information at call/return must be tested.
- Use the program specification as a checklist for the desired behaviour.
The standard test-case should showcase most of the desired program behaviour, with a written explanation of how this is demonstrated.
- Since space in a test document is limited (e.g., 500 lines), show one small standard/complete run of program correctness, and then focus on special cases.
In this way, only the specific output for each special case needs to be include, using ellipsis (...) to elide those sections of output that are common with the standard run.
- To show how the program handles all valid and invalid control-flow, it is allowed to force the program behaviour with special values or commented out code.
Document all special code changes as part of the testing explanation.
Documentation Examples
- Sample test documentation (test.txt)
for the file test.cc
-
A Perl
script is available that allows user to create nicely formatted tabular
ASCII text, suitable for test documentation. Ensure that you set the file to
have executable permissions (see chmod), execute it as
./txt2tbl, and that the machine you are running it on has Perl
installed in the directory specified in the first line of the script (or change
the location to match the path -- see which)]
-
Sample program used for test documentation example (test.cc)
Good versus Better Code and Documentation
A zip file containing all of the sample documentation listed below.
- C++
- Concurrent
