Apertium-regtest

This is a proposal for a regression testing system for use in language modules and translation pairs.

See https://github.com/TinoDidriksen/regtest/wiki for examples of what using it might look like in practice.

Overview

The regression testing system will run a corpus through a pipeline (whether analysis or translation or whatever) recording the output of each step. These outputs will be compared to the expected outputs and an optional list of ideal outputs.

If the actual output is different from the expected or the ideal, this counts as a failing test.

Differences are presented to the developer who can choose to accept some or all of the actual output as the new expected output.

In this way, the regression tests help to ensure that changes to the system do not cause anything to get worse and that the test data is an accurate reflection of the current state of the system while minimizing the effort required of the developer to keep things up-to-date.

Specification

The test runner can be run in either static mode (which functions as a test that can pass or fail) or in interactive mode (which updates the data to reflect the state of the translator).

The test runner will by default check for a file named tests/tests.yaml. This file will contain one or more entries of the form

[name]:
  mode: [mode-name]
  input: [file-name]

Where name is the name of this corpus, mode-name names a pipeline mode (usually abc-xyz or xyz-abc), and the value of input: is a text file where each line contains an input sentence.

The mode will be read from modes.xml and each step will be named in the same fashion as gendebug="yes". That is, using debug-suff is present, otherwise trying to guess a standard suffix, and finally falling back to NAMEME. If more than one step has the same debug suffix, they will be numbered sequentially.

If the input file is not intended to be passed through the entire pipeline, the option start-step: [suffix] can be added, where suffix is the suffix of one of the steps in the pipeline.

If the test does not correspond to any pipeline in modes.xml, mode: [mode-name] can be replaced with command: [cmd] where cmd is an arbitrary bash command which will be run in the main directory of the repository. For the purposes of expected.txt and gold.txt, this will be treated as a pipeline containing a single step named all.

For each step, the test runner will check for files named [name].[step-name].expected.txt and [name].[step-name].gold.txt in the same directory as the input file.

expected.txt is assumed to be the output of a previous run and gold.txt is assumed to be the ideal output. gold.txt can contain multiple ideal outputs for each line, separated by tabs.

In static mode, if the output of a step does not appear in either expected.txt or gold.txt, the test fails.

In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.

See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface may also be available.

Repository Structure

If the test runner does not find a file named tests/tests.yaml, it will guess that the tests live in tests-[name] (for example tests-eng) and offer to clone that repository if it exists.

Conversion Process

Any existing tests will be converted to this format with their inputs placed in an input file and their outputs in the appropriate gold.txt. All expected.txt files will be filled in with the current output of the pipeline.

Any test which does not correspond to an existing mode or the beginning of an existing mode will use command.

In any monolingual module for which I cannot find existing, I will select a few random forms from the analyzer as the corpus. In translation pairs, I will take a few sentences from other pairs which use the same languages.