TODO: This chapter is partially outdated and has to be revised
In this chapter we describe several tests which should be applied regularly to the development and release candidate versions of GAP in the hope to reduce the number of bugs and potential other problems (such as conflicts between the main GAP system and packages) in the released version.
These tests cover GAP computations, in the sense that actual output is compared with expected output, as well as formal aspects of the documentation.
.tst-filesThe directory tst in the GAP root directory contains directories, each of which contains files whose names have the suffix .tst. The format of each of these files must be suitable for being read with the function Test, see section Reference: Test Files. If functionality from GAP packages is needed in a tst file then it is recommended to execute these tests only if the packages are really available, and to load the packages explicitly, for example as follows.
gap> if LoadPackage( "ctbllib" ) <> fail then > if Irr( CharacterTable( "WeylD", 4 ) )[1] <> > [ 3, -1, 3, -1, 1, -1, 3, -1, -1, 0, 0, -1, 1 ] then > Print( "problem with Irr( CharacterTable( \"WeylD\", 4 ) )[1]\n" ); > fi; > fi;
Moreover, it is recommended to group tests that require packages to be performed after all other tests are completed. The directory tst/testinstall contains a subset of tests which are recommended after installing GAP, see https://www.gap-system.org/Download/INSTALL. The idea is that the examples in these .tst files are typical GAP computations, and that running these tests requires only a few minutes. The GAP script tst/testinstall.g will run the tests in tst/testinstall.
Further tests are contained in tst/teststandard. The script tst/teststandard.g will run all tests in both tst/teststandard and tst/testinstall.
The reproducibility of the output of examples in the main GAP manuals (the two manuals in the directories ref and tut) is promised in section Tutorial: Starting and Leaving GAP, under the condition that all examples of a manual chapter are run immediately after starting GAP. Section GAPDoc: Testing Manual Examples contains another paragraph about the use of manual examples for testing purposes.
There is no guarantee that package manuals behave this way. It is up to the package authors whether they want to promise something similar for their package.
Besides tests of the functionality of packages, we are also interested in the effects of packages on the functionality of the main system. The output that is shown in examples in the GAP manuals as well as the output that is prescribed in the test files in tst should be the same when GAP is run without packages or with all packages available. Therefore, the tests in the standard test suite (see 5.6) are run several times, in a GAP that loads no packages and in a GAP that loads all available packages and/or all available autoloaded packages.
Concerning tests of package functionality, in principle the package authors are responsible for providing and running tests for their packages. However, it makes sense to provide test files analogous to those described in section 5.1.
If the record in the PackageInfo.g file of the package (see Reference: The PackageInfo.g File) contains the component TestFile then the file given by the value is read as a part of the regular GAP test suite, see section 5.6. So this file should contain Test statements for suitable files that are part of the package distribution. Since these tests are intended to be run regularly, they should require not more than a few hours of CPU time. (Of course also longer examples can make sense but they should not be listed in the file given by the TestFile component.)
Note that the inclusion of the above package tests refers to the latest released version of each package, not to its development version.
Since the format of the documentation for a package is not prescribed, there is no generic tool for checking manual examples of packages. One possibility to include the examples from package manuals in the standard tests is to create a .tst file with these examples from the manual. (For one implementation of this approach, see the script etc/makexpl in the ctbllib package. It produces a file tst/docxpl.tst in the package directory, containing all examples from the package manual.)
This is still to be provided (and then documented here). Perhaps the easiest way is a check of cross-references in and between the HTML versions of the manuals; and perhaps this should be viewed as a part of the more general task to check the validity of (in particular cross-references in) the GAP website.
(Possible ingedients are doc/test/CheckBooks.g and dev/LinksOfAllHelpSections.g.)
For the manuals in the formats given by the GAPDoc package, multiply defined labels and non-resolved references will be reported at the manual build stage.
Also this is still to be provided (and then documented here).
The standard test suite is given by several targets in the Makefile in the GAP root directory. (So the file Makefile.in is relevant for the maintenance.) It consists of
The tests contained in tst/testinstall, see 5.1,
All tests in all subdirectories of tst, see 5.1,
the tests of the main manuals described in section 5.2,
the package tests described by the TestFile components of PackageInfo.g files of all accepted or deposited packages, and
the tests of packages loading with LoadPackage (Reference: LoadPackage) and LoadAllPackages (Reference: LoadAllPackages)
As is stated in Section 5.3, each test is run at least twice, once in a GAP that loads no packages and then in a GAP that loads all available packages and/or all available autoloaded packages.
Before running these tests, you should make sure that the system is up to date, in particular,
compile package executables where applicable,
call make doc in the root directory, in order to process the main documentation,
Then proceed as follows in the GAP root directory.
Call dev/ci.sh testinstall. This reads the file tst/testinstall.g, which should require just a few minutes.
Call dev/ci.sh testmanuals. This runs the examples in the main GAP manuals, and should also require only a few minutes. (Note that a new GAP process is started for each manual chapter. In order to accelerate this, a GAP workspace is created in the beginning, is reused for each chapter, and removed in the end.)
Call dev/ci.sh teststandard. This runs the tests in tst, and may require a couple of hours.
Call dev/ci.sh testpackages. This starts a new GAP session for each accepted or deposited package, and reads the file given by the component TestFile in the PackageInfo.g file (if this is available). It may require several hours.
Call dev/ci.sh testpackagesloading. This loads each accepted or deposited package in a new GAP session, then starts a new GAP session and calls LoadAllPackages (Reference: LoadAllPackages) to check that all packages may be loaded simultaneosuly. It require just a few minutes.
The output is written to files in the directory dev/log, which does not belong to the distribution of GAP. The file names start with the name of the target (testinstall, teststandard, etc.), followed by 1 for the case that no packages are loaded before the tests, by 2 if all packages are loaded before the tests, and by A if only autoloaded packages are loaded before the tests, followed by the date and time when the test was started, in the format _YY_MM_DD_HH_MM. For the package tests, the output for each package in question is written to a file of its own, and the package name is appended to the file name.
Note that in the case that no package is loaded initially, some packages may be loaded during the tests, due to LoadPackage (Reference: LoadPackage) statements in the test files. So a list of all packages that are loaded at the end of the tests can be found at the end of the output files.
After the tests, please inspect the output files and deal with the differences that were found. The following situations may be expected.
There are differences between the prescribed output and the actual output. This may be the result of code improvements and thus intentional; then the prescribed output should be replaced by the actual one. It may be a side-effect of code changes, for example new or changed methods may choose different generators for a group or a different ordering of conjugacy classes; then one should make sure that the changed output is not caused by worse methods being used than before. If the prescribed and the actual value are essentially different (e.g., true vs. false or different numerical values) then probably we have a serious problem n the sense that (at least) one of the two values is wrong.
There are differences between the results of a test without packages and the same test with packages loaded. It may be that a package provides a better solution than the one that was expected when the input example had been prepared; in this case, it is perhaps advisable to ask the maintainers of the relevant code to modify the example in question, since it is apparently no longer typical. It may also be that a new package version breaks functionality; in this case, the package authors should be informed.
A more subtle difference is a change in the runtime, which is listed more or less detailed in the output files. If the runtime increases w.r.t. an earlier GAP version then this should be reported to the GAP group. If the runtime with packages is substantially bigger than without packages then the package authors should be informed.
When test input is adjusted to the current behaviour, it should be adjusted for the situation that no packages are loaded. (In the case of test input for a package, this means that only the needed packages are loaded, see the component Dependencies.NeededOtherPackages in the file PackageInfo.g.) Note that testing the changed behaviour introduced by another package would be an issue for tests that belong to this package.
The following additional issues are related to testing when a new version of GAP is to be released.
Run all tests.
Add a paragraph about general tests for features: When new documentation is added, add meaningful examples to this documentation, from which users get an idea what typical uses are. In a corresponding file in the tst directory, add examples which cover also pathological input such as empty lists. Add also interesting computations to the tests which are perhaps to be excluded from the regular testing; for features in a package, perhaps distinguish between the tests that are covered by the TestFile component of the PackageInfo.g file and hard tests.
generated by GAPDoc2HTML