The test runner can be run in either static mode (which functions as a test that can pass or fail) or in interactive mode (which updates the data to reflect the state of the translator).
The test runner will by default check for a file named
tests/tests.yaml. This file will contain one or more entries of the form
[name]: mode: [mode-name] input: [file-name]
[name]: command: [command] input: [file-name]
name is the name of this corpus,
mode-name names a pipeline mode (usually
xyz-abc), and the value of
input: is a text file where each line contains an input sentence.
The mode will be read from
modes.xml and each step will be named in the same fashion as
gendebug="yes". That is, using
debug-suff is present, otherwise trying to guess a standard suffix, and finally falling back to
NAMEME. If more than one step has the same debug suffix, they will be numbered sequentially.
command is supplied instead of
mode, the value will be interpreted as a shell command and the only step name will be
For each step, the test runner will check for files named
[name].[step-name].gold.txt in the same directory as the input file.
expected.txt is assumed to be the output of a previous run and
gold.txt is assumed to be the ideal output.
gold.txt can contain multiple ideal outputs for each line, separated by tabs.
In static mode, if the output of a step does not appear in either
gold.txt, the test fails.
In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.
See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface may also be available.
If the test runner does not find a file named
tests/tests.yaml, it will guess that the tests live in
tests-[name] (for example
tests-eng) and offer to clone that repository if it exists.
Any existing tests will be converted to this format with their inputs placed in an input file and their outputs in the appropriate
expected.txt files will be filled in with the current output of the pipeline.
Any test which does not correspond to an existing mode or the beginning of an existing mode will use