Apertium-regtest - Revision history

Trondtr: path typo

2023-06-06T12:17:37Z

path typo

Popcorndude: /* Other */ document sort option

2022-06-08T19:11:36Z

Other: document sort option

Popcorndude: /* Terminology */ clarify usage of "corpus"

2021-11-05T13:59:51Z

Terminology: clarify usage of "corpus"

Popcorndude: /* Specification */

2021-11-05T13:54:01Z

Specification

Popcorndude: hopefully clearer instructions for users

2021-11-05T13:27:27Z

hopefully clearer instructions for users

Popcorndude: link to structure conversion script

2021-08-10T22:21:18Z

link to structure conversion script

Popcorndude: directory structure

2021-08-10T22:13:38Z

directory structure

Popcorndude: /* Overview */

2021-07-24T00:23:16Z

Overview

Popcorndude: deleting line numbers no longer required

2021-07-24T00:02:40Z

deleting line numbers no longer required

Popcorndude: /* Conversion Process */

2021-07-23T23:46:54Z

Conversion Process

@@ Line 66: / Line 66: @@
 Dynamic mode begins the same way, but instead of failing when an output doesn't match <code>expected</code>, it presents those outputs to the user. If the new output is more correct than what is in <code>expected</code>, the user can accept it, in which case it becomes the new <code>expected</code>. On the other hand, if the output has gotten worse, the user can reject it, in which case the test is still failing and the rules in the pipeline need to be fixed.
-The test runner will by default check for a file named <code>tests/tests.json</code>. This file will contain one or more entries of the form
+The test runner will by default check for a file named <code>test/tests.json</code>. This file will contain one or more entries of the form
 <pre>

@@ Line 118: / Line 118: @@
 An individual input line is considered a passing test if it either matches expected or appears in gold for each of the relevant steps and failing otherwise. By default, only the final step of the pipeline is considered relevant. A list of relevant steps can be provided by setting <code>"relevant": [suffixes...]</code> (for example, <code>"relevant": ["morph", "transfer", "postgen"]</code>).
+For modes which output [[Apertium stream format]], it is often desirable to ignore differences in ordering so that, for example, <code>^runs/run&lt;n&gt;&lt;sg&gt;/run&lt;v&gt;&lt;pres&gt;&lt;p3&gt;&lt;sg&gt;$</code> and <code>^runs/run&lt;v&gt;&lt;pres&gt;&lt;p3&gt;&lt;sg&gt;/run&lt;n&gt;&lt;sg&gt;$</code> will be counted as matching. This can be done by setting either <code>"sort": true</code> to apply sorting to all modes in pipeline, or by providing a list such as <code>"sort": ["morph", "biltrans"]</code>.
 In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.

@@ Line 28: / Line 28: @@
 | <code>gold</code> || ideal outputs || We want what's in <code>expected</code> to be as close to one of these as possible.
 |}
+A "corpus" for Apertium-regtest can be any input to one of the translation or analysis pipelines that a repository defines. It will normally be a list of words or sentences, but it can also be analyses (such as when testing the generator).
 === Typical Workflow ===

@@ Line 58: / Line 58: @@
 == Specification ==
+The test runner has 2 modes: static and dynamic.
-The test runner can be run in either static mode (which functions as a test that can pass or fail) or in interactive mode (which updates the data to reflect the state of the translator).
+Static mode runs the input corpus through the pipeline and compares it against <code>expected</code>. If any of the outputs don't match, the test fails.
+Dynamic mode begins the same way, but instead of failing when an output doesn't match <code>expected</code>, it presents those outputs to the user. If the new output is more correct than what is in <code>expected</code>, the user can accept it, in which case it becomes the new <code>expected</code>. On the other hand, if the output has gotten worse, the user can reject it, in which case the test is still failing and the rules in the pipeline need to be fixed.
 The test runner will by default check for a file named <code>tests/tests.json</code>. This file will contain one or more entries of the form

@@ Line 12: / Line 12: @@
 In this way, the regression tests help to ensure that changes to the system do not cause anything to get worse and that the test data is an accurate reflection of the current state of the system while minimizing the effort required of the developer to keep things up-to-date.
+=== Terminology ===
+There are 4 types of corpus files that Apertium-regtest deals with:
+{|class="wikitable"
+! name || description || comments
+|-
+| <code>input</code> || text to be passed through the pipeline || This is a simple text file with one entry per line.
+|-
+| <code>output</code> || temporary files containing the current output of the pipeline || This is what each step of the pipeline output the last time you ran <code>apertium-regtest</code>. You can ignore these files.
+|-
+| <code>expected</code> || the output of the pipeline as of the last commit || This is what each step of the pipeline output the last time someone checked the output and confirmed that it hadn't gotten any worse.
+|-
+| <code>gold</code> || ideal outputs || We want what's in <code>expected</code> to be as close to one of these as possible.
+|}
 === Typical Workflow ===
@@ Line 22: / Line 38: @@
 # accept changes
 # commit changed test files along with dictionaries, transfer rules, etc.
+=== Adding a New Test Corpus ===
+# create a new file under <code>test/</code>
+# put some words, sentences, or whatever in that file
+# add that input file to <code>test/tests.json</code>
+# re-run <code>apertium-regtest</code>
+# add the generated files to git
+Example: If we're trying to add a test for <code>fra-oci_gascon</code>, the input file could be <code>test/fra-oci_gascon-input.txt</code> then we could add the following to <code>test/tests.json</code>:
+<pre>
+"fra-oci_gascon": {
+  "input": "fra-oci_gascon-input.txt",
+  "mode": "fra-oci_gascon"
+}
+</pre>
 == Specification ==
@@ Line 29: / Line 62: @@
 The test runner will by default check for a file named <code>tests/tests.json</code>. This file will contain one or more entries of the form
+<pre>
- {[name]:
-   "mode": [mode-name]
+[name]: {
-   "input": [file-name]}
+  "mode": [mode-name]
+  "input": [file-name]
+}
+</pre>
 === Input Corpus ===
@@ Line 53: / Line 89: @@
 Flat mode is the default, but nested can be specified by adding the following to <code>test/tests.json</code>:
+<pre>
- "settings": {
+"settings": {
-   "structure": "nested"
+  "structure": "nested"
- }
+}
+</pre>
 A flat directory can be automatically converted to a nested one using [https://github.com/apertium/apertium-regtest/blob/master/tools/flat2nest.py this script].
 {|class="wikitable"
-! name || description || flat filename || nested filename
+! name || flat filename || nested filename
 |-
-| input || text to be passed through the pipeline || specified in <code>test/tests.json</code> || specified in <code>test/tests.json</code>
+| input || specified in <code>test/tests.json</code> || specified in <code>test/tests.json</code>
 |-
-| output || temporary files containing the current output of the pipeline || <code>test/[corpus].[step].output.txt</code> || <code>test/output/[corpus].[step].txt</code>
+| output || <code>test/[corpus].[step].output.txt</code> || <code>test/output/[corpus].[step].txt</code>
 |-
-| expected || the output of the pipeline as of the last commit || <code>test/[corpus].[step].expected.txt</code> || <code>test/expected/[corpus].[step].txt</code>
+| expected || <code>test/[corpus].[step].expected.txt</code> || <code>test/expected/[corpus].[step].txt</code>
 |-
-| gold || ideal outputs || <code>test/[corpus].[step].gold.txt</code> || <code>test/gold/[corpus].[step].txt</code>
+| gold || <code>test/[corpus].[step].gold.txt</code> || <code>test/gold/[corpus].[step].txt</code>
 |}
@@ Line 91: / Line 129: @@
 <pre>
 {
-    "general": {
+   "general": {
-        "mode": "wad-tagger",
+       "mode": "wad-tagger",
-        "input": "general-input.txt"
+       "input": "general-input.txt"
-    }
+   }
 }
 </pre>

@@ Line 37: / Line 37: @@
 Where <code>name</code> is the name of this corpus, <code>mode-name</code> names a pipeline mode (usually <code>abc-xyz</code> or <code>xyz-abc</code>), and the value of <code>"input":</code> is a text file where each line contains an input sentence. Line breaks can be included in the input by writing <code>\n</code> and comments beginning with <code>#</code> will be ignored.
-=== Mode Specification
+=== Mode Specification ===
 The mode will be read from <code>modes.xml</code> and each step will be named in the same fashion as <code>gendebug="yes"</code>. That is, using <code>debug-suff</code> is present, otherwise trying to guess a standard suffix, and finally falling back to <code>NAMEME</code>. If more than one step has the same debug suffix, they will be numbered sequentially.
@@ Line 45: / Line 45: @@
 If the test does not correspond to any pipeline in <code>modes.xml</code>, <code>mode: [mode-name]</code> can be replaced with <code>"command": [cmd]</code> where <code>cmd</code> is an arbitrary bash command which will be run in the main directory of the repository. For the purposes of <code>expected.txt</code> and <code>gold.txt</code>, this will be treated as a pipeline containing a single step named <code>all</code>.
-=== Other ===
+=== Directory Structure ===
+There are 4 types of files associated with a particular corpus: input, output, expected, and gold. There are two arrangements of files currently supported: flat and nested.
-For each step, the test runner will check for files named <code>[name].[step-name].expected.txt</code> and <code>[name].[step-name].gold.txt</code> in the same directory as the input file.
+In flat mode, all files are placed in the same directory. In nested mode, output, expected, and gold each have a separate subdirectory within <code>test/</code>.
-<code>expected.txt</code> is assumed to be the output of a previous run and <code>gold.txt</code> is assumed to be the ideal output. <code>gold.txt</code> can contain multiple ideal outputs for each line.
+Flat mode is the default, but nested can be specified by adding the following to <code>test/tests.json</code>:
+ "settings": {
+   "structure": "nested"
+ }
+{|class="wikitable"
+! name || description || flat filename || nested filename
+|-
+| input || text to be passed through the pipeline || specified in <code>test/tests.json</code> || specified in <code>test/tests.json</code>
+|-
+| output || temporary files containing the current output of the pipeline || <code>test/[corpus].[step].output.txt</code> || <code>test/output/[corpus].[step].txt</code>
+|-
+| expected || the output of the pipeline as of the last commit || <code>test/[corpus].[step].expected.txt</code> || <code>test/expected/[corpus].[step].txt</code>
+|-
+| gold || ideal outputs || <code>test/[corpus].[step].gold.txt</code> || <code>test/gold/[corpus].[step].txt</code>
+|}
+=== Other ===
-An individual input line is considered a passing test if it appears in either <code>expected.txt</code> or <code>gold.txt</code> for each of the relevant steps and failing otherwise. By default only the final step of the pipeline is considered relevant. A list of relevant steps can be provided by setting <code>"relevant": [suffixes...]</code> (for example, <code>"relevant": ["morph", "transfer", "postgen"]</code>).
+An individual input line is considered a passing test if it either matches expected or appears in gold for each of the relevant steps and failing otherwise. By default, only the final step of the pipeline is considered relevant. A list of relevant steps can be provided by setting <code>"relevant": [suffixes...]</code> (for example, <code>"relevant": ["morph", "transfer", "postgen"]</code>).
 In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.
-See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface may also be available.
+See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface is also be available.
 == Repository Structure ==

@@ Line 97: / Line 97: @@
 Like the expected output, gold output is delimited and sorted by hash. Multiple possible ideals are separated by <code>[/option]</code>.
-Since the entries are delimited by hashes, the recommended way to interact with these files is via the provided interfaces. However, there are instances where you might want to add values directly. For small corpora, you can copy the corresponding <code>expected.txt</code> file and edit the entries (note that you need to remove the <code>#0</code> after the hashes<ref>In <code>output.txt</code> files, this is the line number of the input (for sorting purposes). <code>expected.txt</code> use the same format, but set the line number to 0, since it doesn't matter.</ref>). For larger corpora there are some conversion scripts available at https://github.com/apertium/apertium-regtest/tree/master/tools. Bug [[User:Popcorndude]] if you need more of these.
+Since the entries are delimited by hashes, the recommended way to interact with these files is via the provided interfaces. However, there are instances where you might want to add values directly. For small corpora, you can copy the corresponding <code>expected.txt</code> file and edit the entries. For larger corpora there are some conversion scripts available at https://github.com/apertium/apertium-regtest/tree/master/tools. Bug [[User:Popcorndude]] if you need more of these.
 <pre>
@@ Line 112: / Line 112: @@
 In any module for which I cannot find existing tests, I will select a few random forms from the analyzer as the corpus. <code>gold.txt</code> will be left empty.
-== Footnotes ==
-<references/>
 [[Category:Quality_control]]