View source for User:Popcorndude/Unit-Testing

Proposed file structure for unit testing framework.

== Unit-Testing Specification ==

<code>make test</code> will check for a file named <code>tests/tests.yaml</code> which can include other files in the same directory.

At the top level of a test file, <code>include</code> is a list of included files (paths given relative to the directory of the current file). All other keys are names of tests.

Each test consists of a collection of pairs of strings which can be written as <code>left: right</code> or placed in a TSV file and referenced with <code>tsv-file: [path]</code>. Multiple valid outputs can be written as a list

 left:
   - right1
   - right2

or by having more than 2 columns in the TSV file.

Each test also specifies what mode to run. This specification can be either <code>lr</code> (left side is input) or <code>rl</code> (right side is input). Bidirectional tests may be created by listing both.

 [direction]:
   mode: [mode-name]
   match: [match-mode]
   options: [option-string]
   stream-type: [type]

Options for <code>match-mode</code> are documented below. The default value if not specified is <code>one-of</code>.

<code>option-string</code> will be passed to the <code>apertium</code> executable. It defaults to <code>-f none</code>.

Options for <code>stream-type</code> are documented below. This will usually be inferred by the test runner and so need not be specified.

If no settings other than <code>mode</code> are specified, this can be abbreviated to <code>[direction]: [mode-name]</code>.

=== Matching Modes ===

{| class="wikitable"
! mode name !! description !! reversal
|-
| <code>exact</code> || pipeline output and right side must be identical || pipeline output and left side must be identical
|-
| <code>one-of</code> || pipeline output must appear on the right side || pipeline output of one of the right sides must match the left side
|-
| <code>include</code> || right side must appear in the pipeline output ||
|-
| <code>exclude</code> || right side must not appear in the pipeline output ||
|}

=== Stream Types ===

{| class="wikitable"
! stream type !! description
|-
| <code>text</code> || values are compared as strings
|-
| <code>readings</code> || values with the same surface or source (preceding the first <code>/</code>) are compared based on remaining readings (unordered)
|-
| <code>no-src</code> || like <code>readings</code>, but without surface form
|-
| <code>anaphora</code> || like <code>readings</code>, but the last reading is matched separately
|}

If the value in the text file contains any character in <code>^/$</code>, interpretation will default to <code>readings</code>. Otherwise <code>text</code> will be used. LUs not delimited by <code>^$</code> will be split on spaces.

<code>*</code> will match arbitrary readings and <code>?</code> will match a single arbitrary reading. These can be escaped with <code>\</code>.

Under <code>readings</code> LUs in corresponding positions will be considered non-matching if they have different first segments.

== Unit-Testing Example Files ==
<code>apertium-eng-spa/tests/tests.yaml</code>
 include:
   - other_file_1.yaml
   - other_file_2.yaml
 "possession":
   lr: eng-spa
   rl: spa-eng
   "the cat's box": "la caja del gato"
   "my sister's socks": "los calcetines de me hermana"
 "noun/verb disam":
   lr:
     mode: eng-spa-tagger
     match: exact
   "the cat's box": "the/the<det><def><sp> cat's/cat<n><sg>+'s<gen> box/box<n><sg>"
 "de":
   lr:
     mode: spa-eng
     match: one-of
   "el gato de mi hermana":
     - "my sister's cat"
     - "the cat of my sister"

<code>apertium-eng/tests/tests.yaml</code>
 "past tense":
   lr:
     mode: eng-morph
     match: include
   rl:
     mode: eng-gener
     match: exact
   tsv-file: past-tense-tests.tsv
 "disam":
   lr:
     mode: eng-tagger
     match: exclude
   "to be purple": "to/ be/be<vbser><imp> purple/"

<code>apertium-eng/tests/past-tense-tests.tsv</code>
 sang	sing<vblex><past>
 jumped	jump<vblex><past>

== Annotated Unit-Testing Example Files ==
<code>apertium-eng-spa/tests/tests.yaml</code>
 include:     # run the tests in these files as well
   - other_file_1.yaml
   - other_file_2.yaml
 "possession":       # test named "possession"
   lr: eng-spa       # left side  | apertium eng-spa => right side
   rl: spa-eng       # right side | apertium spa-eng => left side
   "the cat's box": "la caja del gato"
   "my sister's socks": "los calcetines de me hermana"
 "noun/verb disam":  # test named "noun/verb disam"
   lr:               # only has 1 direction
     mode: eng-spa-tagger
     match: exact    # output must match what we've written below exactly
                     # since there's only one thing on the right, this is
                     # equivalent to the default behavior,
                     # so we're just being explicit
   "the cat's box": "^the/the<det><def><sp>$ ^cat's/cat<n><sg>+'s<gen>$ ^box/box<n><sg>$"

<code>apertium-eng/tests/tests.yaml</code>
 "past tense":
   lr:
     mode: eng-morph
     match: include   # the right side of the test must appear in the output
                      # but the test will still pass if other things appear as well
   rl:
     mode: eng-gener
     match: exact
   tsv-file: past-tense-tests.tsv  # read the test data from a tab-separated list
 "disam":
   lr:
     mode: eng-tagger
     match: exclude   # the output can contain other things, but must not contain
                      # the readings listed
   "to be purple": "^to/$ ^be/be<vbser><imp>$ ^purple/$"
 "multiwords":
   lr: eng-morph
                      # we need ^ and $ here because there are spaces
   "good morning": "^good morning/good morning<ij>$"

== Corpus Regression Testing ==

The test runner can be run in either static mode (which functions as a test that can pass or fail) or in interactive mode (which updates the data to reflect the state of the translator).

The test runner will by default check for a file named <code>tests/regressions.yaml</code>. This file will contain one or more entries of the form

 [name]:
   mode: [mode-name]
   input: [file-name]

Where <code>name</code> is the name of this corpus, <code>mode-name</code> names a pipeline mode (usually <code>abc-xyz</code> or <code>xyz-abc</code>), and the value of <code>input:</code> is a text file where each line contains an input sentence.

The mode will be read from <code>modes.xml</code> and each step will be named in the same fashion as <code>gendebug="yes"</code>. That is, using <code>debug-suff</code> is present, otherwise trying to guess a standard suffix, and finally falling back to <code>NAMEME</code>. If more than one step has the same debug suffix, they will be numbered sequentially.

For each step, the test runner will check for files named <code>[name].[step-name].expected.txt</code> and <code>[name].[step-name].gold.txt</code> in the same directory as the input file.

<code>expected.txt</code> is assumed to be the output of a previous run and <code>gold.txt</code> is assumed to be the ideal output. <code>gold.txt</code> can contain multiple ideal outputs for each line, separated by tabs.

In static mode, if the output of a step does not appear in either <code>expected.txt</code> or <code>gold.txt</code>, the test fails.

In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.

See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface may also be available.

== Repository Structure ==

If the test runner does not find a directory named <code>tests</code> containing the expected YAML files, it will guess that the tests live in <code>tests-[name]</code> (for example <code>tests-eng</code>) and offer to clone that repository if it exists.

== Scraping Tests ==

Standard scripts will be available to construct test files from wiki pages and comments in source files.

Existing wiki tests will be reformatted to use the [[Template:TransferTest]] template (since [[Template:Test]] doesn't specify the target language and I plan to double check all the formatting anyway to catch formatting errors or plaintext tests).

{{TransferTest|abc|xyz|sentence 1|sentence 2}}

 <nowiki>{{TransferTest|abc|xyz|sentence 1|sentence 2}}</nowiki>

Will be converted to

 "[nearest header]":
   lr: abc-xyz
   "sentence 1": "sentence 2"

Templates for other sorts of tests are discouraged, but can be added if desired.

== Testvoc ==

This project could probably be expanded to include trimmed testvoc, though I currently don't understand the existing code well enough to propose anything concrete.