User:Popcorndude/Unit-Testing - Revision history

Popcorndude at 16:40, 2 March 2021

2021-03-02T16:40:21Z

Popcorndude: /* Scraping Tests */

2021-03-02T16:14:33Z

Scraping Tests

Popcorndude: /* Scraping Tests */

2021-03-02T16:14:01Z

Scraping Tests

Popcorndude: /* Corpus Regression Testing */ link to example

2021-03-02T15:53:15Z

Corpus Regression Testing: link to example

Popcorndude at 15:48, 2 March 2021

2021-03-02T15:48:11Z

Popcorndude: note about tests in a separate repo

2021-03-02T02:24:58Z

note about tests in a separate repo

Popcorndude at 02:20, 2 March 2021

2021-03-02T02:20:15Z

Show changes

Popcorndude at 15:37, 24 February 2021

2021-02-24T15:37:40Z

Popcorndude: /* TODO */

2021-02-24T15:23:08Z

TODO

Popcorndude: todo list

2021-02-24T02:22:40Z

todo list

@@ Line 205: / Line 205: @@
 This project could probably be expanded to include trimmed testvoc, though I currently don't understand the existing code well enough to propose anything concrete.
+== Conversion Process ==
+Any existing tests which have text as input will be converted to regression tests and their expected outputs will be included as gold outputs and the <code>expected.txt</code> files will be filled in with the current output of the pipeline.
+Any tests which take stream format as input will be converted to unit tests.

@@ Line 199: / Line 199: @@
 Or, in XML:
- <!-- TEST abc-xyz: sentence 1 => sentence 2
+ <nowiki><!-- TEST abc-xyz: sentence 1 => sentence 2
- -->
+ --></nowiki>
 == Testvoc ==

@@ Line 192: / Line 192: @@
 Templates for other sorts of tests are discouraged, but can be added if desired.
+The equivalent source file comment to the above will be
+ # TEST abc-xyz: sentence 1 => sentence 2
+Or, in XML:
+ <!-- TEST abc-xyz: sentence 1 => sentence 2
+ -->
 == Testvoc ==

@@ Line 168: / Line 168: @@
 In dynamic mode, differences between the output and the files will be presented to the user, who will have the option to add the output to either file.
+See https://github.com/TinoDidriksen/regtest/wiki for images of what the workflow in dynamic mode might look like. A command line interface may also be available.
 == Repository Structure ==

@@ Line 1: / Line 1: @@
 Proposed file structure for unit testing framework.
-== TODO ==
-* testvoc?
-* wiki format? [[Template:Test]], [[Template:TransferTest]], and plain text seem to be the current options (with the last doubtless hiding many complexities)
-** [[Template:TransferTest]] is only used by eng-kir while [[Template:Test]] appears on almost 200 pages
-** do we want wiki format for more than just full-pipeline tests?
 == Unit-Testing Specification ==
-<code>make test</code> will check for a file named <code>tests/apertium-[name].tests.yaml</code> which can include other files in the same directory.
+<code>make test</code> will check for a file named <code>tests/tests.yaml</code> which can include other files in the same directory.
 At the top level of a test file, <code>include</code> is a list of included files (paths given relative to the directory of the current file). All other keys are names of tests.
@@ Line 158: / Line 151: @@
 The test runner can be run in either static mode (which functions as a test that can pass or fail) or in interactive mode (which updates the data to reflect the state of the translator).
-The test runner will by default check for a file named <code>tests/apertium-[name].regressions.yaml</code>. This file will contain one or more entries of the form
+The test runner will by default check for a file named <code>tests/regressions.yaml</code>. This file will contain one or more entries of the form
  [name]:
@@ Line 178: / Line 171: @@
 == Repository Structure ==
-If the test runner does not find a directory named <code>tests</code> containing the expected YAML files, it will guess that the tests live in <code>[repo-name]-tests</code> (for example <code>apertium-eng-tests</code>) and offer to clone that repository if it exists.
+If the test runner does not find a directory named <code>tests</code> containing the expected YAML files, it will guess that the tests live in <code>tests-[name]</code> (for example <code>tests-eng</code>) and offer to clone that repository if it exists.
+== Scraping Tests ==
+Standard scripts will be available to construct test files from wiki pages and comments in source files.
+Existing wiki tests will be reformatted to use the [[Template:TransferTest]] template (since [[Template:Test]] doesn't specify the target language and I plan to double check all the formatting anyway to catch formatting errors or plaintext tests).
+{{TransferTest|abc|xyz|sentence 1|sentence 2}}
+ <nowiki>{{TransferTest|abc|xyz|sentence 1|sentence 2}}</nowiki>
+Will be converted to
+ "[nearest header]":
+   lr: abc-xyz
+   "sentence 1": "sentence 2"
+Templates for other sorts of tests are discouraged, but can be added if desired.
+== Testvoc ==
+This project could probably be expanded to include trimmed testvoc, though I currently don't understand the existing code well enough to propose anything concrete.

@@ Line 7: / Line 7: @@
 At the top level of a test file, <code>include</code> is a list of included files (paths given relative to the directory of the current file). All other keys are names of tests.
-Each test consists of a collection of pairs of strings which can be written as <code>left: right</code> or placed in a TSV file and referenced with <code>tsv-file: [path]</code>.
+Each test consists of a collection of pairs of strings which can be written as <code>left: right</code> or placed in a TSV file and referenced with <code>tsv-file: [path]</code>. Multiple valid outputs can be written as a list
+ left:
+   - right1
+   - right2
+or by having more than 2 columns in the TSV file.
 Each test also specifies what mode to run. This specification can be either <code>lr</code> (left side is input) or <code>rl</code> (right side is input). Bidirectional tests may be created by listing both.
@@ Line 16: / Line 22: @@
    options: [option-string]
-<code>match-mode</code> can be one of <code>exact</code> (output must be exactly as written in the test), <code>include</code> (the values in the test must be present in the output), or <code>exclude</code> (the values in the test must not be present in the output). This defaults to <code>exact</code> if not specified.
+<code>match-mode</code> can be one of <code>exact</code> (output must be exactly as written in the test), <code>include</code> (the values in the test must be present in the output), <code>exclude</code> (the values in the test must not be present in the output), or <code>one-of</code> (the output must be among the listed options). If not specified, this defaults to <code>one-of</code>, which is equivalent to <code>exact</code> is the tests are 1:1.
 <code>option-string</code> will be passed to the <code>apertium</code> executable. It defaults to <code>-f none</code>.
@@ Line 31: / Line 37: @@
 ** [[Template:TransferTest]] is only used by eng-kir while [[Template:Test]] appears on almost 200 pages
 ** do we want wiki format for more than just full-pipeline tests?
+* can 1:n tests have rl?
-* have input map to multiple outputs
 == Example Files ==
@@ Line 48: / Line 54: @@
      match: exact
    "the cat's box": "^the/the<det><def><sp>$ ^cat's/cat<n><sg>+'s<gen>$ ^box/box<n><sg>$"
+ "de":
+   lr:
+     mode: spa-eng
+     match: one-of
+   "el gato de mi hermana":
+     - "my sister's cat"
+     - "the cat of my sister"
 <code>apertium-eng/tests/tests.yaml</code>