The tester manual overview refers to all other sections:
Keywords on the left margin are used to direct testing so that comments can be almost anywhere. Keywords are described below in grouped bulletins.
This unique description helps find tests.
Mulitple line commands can be as many lines as needed to complete the keyword. Note the special section on endings:
Multiple line commands can be ended with:
Multiple lines of commands may have blank lines sprinkled in which are ignored unless it is the run command. Use the -all option to make blank lines significant in all commmands.
Some keywords are limited to one line of information:
See the Test Specification Example, sample for an example.
Define beforeall , afterall and envall commands
Define default groups that have similar characteristics.
Define each product specific test to succeed.
Defining tests includes setting up names, environment variables and any set up and tear down commands that are shared.
Use names to specify command locations that vary from platform to platform. Use names to specify the location of programs to test. Anything that can be moved can be named so that a name file can be used to test out a new environment. This achieves location independence.
The concept of system defined names is discussed later which will allow the tester to refer to the unique names that may change with every test run but stay consistent during the run of an individual test.
Note that a name is used within the test document with a dollar symbol in front of it. If you need to use the dollar symbol in your test document then use a backslash to quote it, i.e. \$DATA.
This sample shows that the environment variable WORLD is set for this test specification.
It must be stated that this environment variable may not be in force for the execution of all tests. It all depends on how you drive the tester. Check out the -queuejobs option which insures that the entire test session is done as one stream of commands.
The -restart option will insure that each test is done by itself so the environment variable for the WORLD would go away even before the first test is executed. Otherwise, use the env command to give a shell variable to a specific test.
And all environment variables are shell specific. In this case, the senenv command applies only to cshell, csh. The bourne shell would look like:
The default shell is the bourne shell so this is the form expected by default. The variable has to be exported to export its value to any subshells.
Even as the tester executes, it may not be actually executing a test. By default the tester just creates a shell to execute a series of tests. See the -execute option to see a way to have it doing things immediately.
In this case global files may be put in place to allow all tests to operate on common data so that expected results are the same each time.
This beforeall keyword can also apply to a default group of tests. In that case the before all commands execute before all tests in that specific group.
This afterall keyword can also apply to a default group of tests. In that case the after all commands execute after all tests in that specific group.
This shows a group for error testing:
The idea of a test group is optional. The test specification will run without defining any specific test group.
The following keywords may be specified as a ``default'' test group:
The afterall and beforeall commands will be executed after and before all tests that follows the definition of this default test group.
Any default keywords apply to each test that follows but may be overridden by any specific test definition using the same keyword. The override of that keyword only applies to the current test. The next test goes back to the default keyword.
This shows an expanded group for error testing:
In this case any test in this group will expect the Standard Error output. This means it will remember the Standard Error output when the test is executed. Errors are often writing to standard error.
The expect keyword can use the following keywords to specify expecting certain output:
Really the heart of testing is the ability to backup these expected results , restore files for checking results and to reproduce results .
The output has to be explicitly put there by the specific test as shown:
An expect file is created by the name of $TESTSPEC .log. The tester logs expect files to compare the results to $TESTSPEC .R_LOG. Any mis-matched data will be found in the $TESTSPEC .log.CMP file.
Tests typically FAIL or SUCCEED but not always. The save keyword will still create a results file but will save it to $TESTSPEC .S_LOG.
This particular test will always succeed because it has no exit or expect conditions. It is still better to check the exit value to verify that the program itself ran successfully as shown:
Now as long as the grep command completes successfully, the output will be written to $TESTSPEC .log and saved to $TESTSPEC .S_LOG. Any expect or save files are automatically cleaned up in this process if the test is successful.
Notice how we have to monkey around with the name to get the expected output to be recognized. Check out the prune keyword for an alternative approach.
The prune keyword allows any number of commands for filtering the expected data before it is processed for success or failure. If the data is generated, it is also pruned before the generated data is saved.
Here is an example of testing the executable of a product without having to change the development process.
This makes use of the binary compare in the diff section as well as an alternate expect ed output.
Another example filters out dates:
This scans for lines of analysis with a scan routine and filters today's date with a dateout.pl script. The ``expect both'' will capture both standard error and standard output from the run command.
The clean up after removes any backup file created by the dateout.pl utility. If the files do not match the previously generated data, then the test will FAIL .
The script delete.pl leaves backup files of the before delete image for each file processed. So the clean up must come along and remove those backup files with the ``rm -f *bak'' command.
The problem with this example is that if $FILESPEC .data is shared by more than one test, the delete.pl routine has just destroyed global data. Look at the before section to see a solution.
Often the development tools like make have commands which support this clean up as well. See the prune section for a sample of ``make clean'' in the after section.
Use the -keep option to skip the after and afterall sections of the current test session. This may help in debugging tests.
At this point the after section is expanded to clean up the test specific data, $TESTSPEC .data. Use the -keep option to skip the after and afterall sections of the current test session. This may help in debugging tests.
Successful tests like the one shown generally return zero.
Tests that check for user errors often return a positive number indicating the error. In this example the return of exit 2 should occur with a -bad option:
Tests that check for operating system errors often return a negative number like this example:
If the exit value varies from the numerical value specified then that specific test will FAIL .
See some greater detail on this in envall .
It helps if these classes are documented in the test specification so that others can use them to run the tests that apply to them. The use of the -class option allows a tester to include tests that refer to that class. The use of the -unclass option allows a tester to exclude tests.
Sample classes might be:
The class keyword specifies class names that apply to all tests that follows the definition of this default test group. If a test also specifies a class then it is in addition to the group class values.
Finally, here are some samples given the test classes mentioned above using -class and -unclass options:
Benchmarking critial tests: tester -x -bench -class critial t.scan
Checking all known failures: tester -x -class kf t.scan
Checking fixes of known bugs: tester -x -class kf+fix t.scan
Run test suite minus bugs: tester -x -class suite -unclass kf t.scan
The following is a list:
Things concerning a group begin with $DEF. See $DEFCLASS and $DEFDESC .
Things concerning a test begin with $TEST. See $TESTCLASS , $TESTCMD , $TESTDESC , $TESTID , $TESTNAME , $TESTNUM , $TESTSPEC .
The tester version can be see with $TESTVER and $TESTVERSION .
The tester command is seen with $TESTSCMD .
The tester runtime directory is resolved at test time and placed in $TESTDIR . This insures all shell output and command output will go to the test subdirectory during complex testing. Also, the tester can always return for error recovery using, cd $TESTDIR .
The description of the test specification can be found at $TESTSDESC found in a specific file called $FILENAME . At test time the name $FILESPEC includes the run time directory.
Errors may come with timestamps, file inode numbers and so forth. But the validity of the error may be just the exit number which proves the error was caught (a successful test). The save command will keep the standard error around as informative information for the tester analysis.
Note that the above test succeeds even though the standard error output changes from run to run. The FAIL check is looking for an exit of the value 3.
The save output is renamed in this case to $TESTSPEC .S_SE. Any save output is renamed using the ``S_'' to distinguish it from expected results , ``R_''.
A user tag may be applied in a test to save user output that will vary.
These are the same keywords shown as default keywords. To override a default put the specific test value after ``test''. This override only applies to the current test.
Tests are fairly simple when set up this way. Show below are two tests with a simple run command.
The run line must be terminated by either another keyword, the end of the file, an end keyword or a comment . In this case we have put the shell omment symbol to insure that the test is run as a single command.
A multiple line command would look like:
This multiple line input will run the ``grep'' command and read in as input all lines up to the keyword end . As you can guess, only the line ``and who can guess'' will be found by grep which looks for any lines with the word ``who'' in it.
Name substitution is allowed anywhere in a test. So this is a test template for searching for generic things with any search routine:
The find routine looks for a word in a file which is names according to the name of the test specification. For more info see $FILESPEC . In this case the end of the file indicates this is a one line command.
When running the -queuejobs options, this is the point where all tests are completed and put together as a single test. In most cases tests are built in chunks and run as they occur as in -execute .
It is the natural plact to put tester notes, to document classes and to discuss any special usage notes for this file.
Any non-keyword in column one is ignored.
And any blank line is also ignored in a test if it falls inside of any multiple line command except the run command.
Isn't the above easier to read than:
Any shell comment is honored using the pound sign in the first column. If you need to pass through shell comments, then use name substitution with a keyword as shown below:
For example, the entry above would substitute:
``regres io''
for this name. This lets the test creator define useful group classes.
Classes are used to select specific tests. It is recommended that these names be documented. If possible adopt names that will be in common use. Like:
regres - regression test for a complete regression suite.
io - input output test.
hog - testing group that takes up all of memory.
For example, the entry above would substitute:
``valid tests''
for this name. This lets the test creator uniquely define test groups.
To test the program ``scan'' one might create a test specification called t.scan. In that case any reference to $FILENAME returns the string ``t.scan''.
In creating a large number of tests it is important to choose some conventions that allow tests and test data to stand out and be unique.
In a test specification called t.scan we see the entry below for clean up.
For example, the entry above would substitute ``t.scan'' for this name. This lets the test creator to remove global data used for testing.
In the next section the $FILESPEC includes the complete test file specification. It is safer to use $FILESPEC which includes $TESTDIR .
So the t.scan file can always be found as $FILESPEC . The test running in /test/scan will resolve to:
/test/scan/t.scan
In complex tests this allows a tester to get the original $FILENAME without trying to keep track of directory movements. A tester can create a common file using $FILESPEC as a root name as shown:
For example, the entry above would substitute:
``regres io write''
for this name. This lets the test creator define useful test classes. Test classes are used to select specific tests. It is recommended that these names be documented. If possible adopt names that will be in common use and test specific. Like:
kf - known failures for outstanding bugs testing..
dangerous - dangerous test which may reboot the system for example.
critical - critical test that must pass for certification.
hog - testing that takes up all of memory.
The test below shows its use to document a test:
For example, the entry above would substitute:
``changing an employee last name''
for this name. This lets the test creator uniquely define tests.
These descriptions can form an outline of the test plan so that the test specification will flow naturally from group to group and test to test.
The above is a typical set of descriptions as a program begins to wring out all possible errors that a user might see.
For complex tests that create output in strange places, this can be used to copy or move the data back to a place where the tester will see it.
Notice that $TESTSPEC and $FILESPEC implicitly use $TESTDIR to know where to place the test data.
An explicit use of it might be:
However, since the test directory is subject to change it is usually not used. Most recording of tests will be done by $TESTNAME or $TESTDESC to avoid having test files that will vary by test location.
tester -class regres -execute -tag foo t.scan
This example shows the identifier ``foo'' that will be substituted for any $TESTID or $TESTNAME within the test specification.
By creating unique tags a tester can reuse the same test file over and over. For example, one file of tests may test out operations across three operating systems. MAC, PC and UNIX would be the unique tags.
$FILENAME + $TESTID + $TESTNUM
So the t.scan file with the identifier, foo on the 5th initial test will be named:
t.scanfoo005
Tests will be labeled from 001 to 999 which shows the limit of 999 tests per test specification. See Test Ordering .
Sometimes it is useful to use the ttags routine to track tests since the numbering of tests can be difficult to follow.
By default this test numbering is preserved in the test order database. See Test Ordering for more information.
$TESTDIR + $FILENAME + $TESTID + $TESTNUM
So the t.scan result file with the identifier, foo, on the 5th test, running in /test/scan will be named:
/test/scan/t.scanfoo005.result
Tests will be labeled from 001 to 999 which shows the limit of 999 tests per test specification.
For example, the entry above would substitute:
``testing the scan program''
This in no way creates version control for the group of tests being run. To do that uses the names feature of the tester language to create your own $Version variable.
This in no way creates version control for the group of tests being run. To do that uses the names feature of the tester language to create your own $Version variable.
The idea of a failure and success is problematic. So the following guidelines apply:
All tests that fail will show up in the $TESTNAME .FAILED file so they should stand out like a sore thumb. A summary of failed tests will show up in the $FILESPEC .result file.
So once more, make all tests succeed when they find the expected results. So the failed tests will be work for the tester to resolve. These tests become known failures. Known failures are managed with fixes. And the process can use the class and -unclass features of the tester to manage these problems.
The tester achieves automation by using a test specification. The coverage is easier to achieve and maintain by automating steps:
And once this exists a benchmark is just another test.