Corpus test - Revision history

Popcorndude at 20:16, 23 July 2021

2021-07-23T20:16:56Z

2017-09-14T14:36:23Z

Seeing the forest in all the trees

2014-10-07T11:44:49Z

Link to French page

2012-03-24T10:40:22Z

Category:Documentation in English

2012-01-15T00:22:41Z

Creation of a corpus

2012-01-15T00:20:15Z

Creation of a corpus

2012-01-05T11:25:15Z

Creation of a corpus

2012-01-05T11:24:26Z

Creation of a corpus

2012-01-05T11:19:11Z

Helper script: diffing while translating

2012-01-05T11:18:34Z

Seeing the forest in all the trees

@@ Line 3: / Line 3: @@
 {{TOCD}}
 '''Corpus testing''' is the way of testing (translating) a whole corpus and compare the result to last time the corpus was translated. This is very useful if you want to change a rule or word and want to get an overview of the consequences on real-life text. Along with [[testvoc]] and [[regression testing]] it is a good way to test that your translator works, and that your changes haven't broken anything.
+Everything listed below can also be done automatically using [[Apertium-regtest]].
 ==Creation of a corpus==

@@ Line 83: / Line 83: @@
 The following command will give you a hitparade (frequency list) of word-based changes in translation output:
 <pre>
-$ diff -U0 origina_traduko.txt nova_traduko.txt | dwdiff --diff-input |grep -v '^@' |sed 's/.*\[-//'|sed 's/+}.*//'  | sort|uniq -c|sort -n
+$ diff -U0 origina_traduko.txt nova_traduko.txt | dwdiff --diff-input |grep -v '^@' |sed 's/.*\[-//'|sed 's/+}.*//'  | sort|uniq -c|sort -nr
 </pre>

@@ Line 1: / Line 1: @@
+[[Test de corpus|En français]]
 {{TOCD}}
 '''Corpus testing''' is the way of testing (translating) a whole corpus and compare the result to last time the corpus was translated. This is very useful if you want to change a rule or word and want to get an overview of the consequences on real-life text. Along with [[testvoc]] and [[regression testing]] it is a good way to test that your translator works, and that your changes haven't broken anything.

@@ Line 106: / Line 106: @@
 [[Category:Quality control]]
 [[Category:Terminology]]
+[[Category:Documentation in English]]

@@ Line 6: / Line 6: @@
 Before you start you first need a [[Corpora|corpus]]. Look in apertium-eo-en/corpa/enwiki.crp.txt.bz2 (run bunzip2 -c enwiki.crp.txt.bz2 > en.crp.txt) to get an idea of how it should look:
-* Use <code>grep</code> to remove all lines in the original corpus containing <code>#</code> and <code>@</code>, these symbols are used in Apertium for marking errors in the bilingual dictionary and transfer.
+* Use <code>grep</code> to remove all lines in the original corpus containing <code>#</code> and <code>@</code> as these symbols are used in Apertium for marking errors in the bilingual dictionary and transfer.
 ** e.g. <code>grep -v '[@#]' original-corpus > clean-corpus</code>
 * Use the command <code>nl -s '. '</code> to number the lines in the corpus.

@@ Line 6: / Line 6: @@
 Before you start you first need a [[Corpora|corpus]]. Look in apertium-eo-en/corpa/enwiki.crp.txt.bz2 (run bunzip2 -c enwiki.crp.txt.bz2 > en.crp.txt) to get an idea of how it should look:
-* Grep out all lines in the original corpus with # and @ – this will help you find real problems in bidix (@) and target language monodix (#).
+* Use <code>grep</code> to remove all lines in the original corpus containing <code>#</code> and <code>@</code>, these symbols are used in Apertium for marking errors in the bilingual dictionary and transfer.
-** eg. <code>grep -v '[@#]' original-corpus > clean-corpus</code>
+** e.g. <code>grep -v '[@#]' original-corpus > clean-corpus</code>
-* Pipe through nl -s '. ' to get the right line numbers.
+* Use the command <code>nl -s '. '</code> to number the lines in the corpus.
-** eg. <code>nl clean-corpus > clean-numbered-corpus</code>
+** e.g. <code>nl clean-corpus > clean-numbered-corpus</code>
 ==testcorpus script==

@@ Line 96: / Line 96: @@
 </pre>
-Call it "mindiff" or something, and you can keep doing <code>mindiff origina_traduko.txt nova_traduko.txt |tail</code> to check on new differences as they come. Or you can do <code>mindiff origina_traduko.txt nova_traduko.txt -U0|tail |dwdiff --diff-input</code> as shown above.
+Call it "mindiff" or something, and you can keep doing <code>mindiff origina_traduko.txt nova_traduko.txt |tail</code> to check on new differences as they come. Or you can do <pre>mindiff origina_traduko.txt nova_traduko.txt -U0|tail |dwdiff --diff-input</pre> as shown above.
 ==Notes==

@@ Line 77: / Line 77: @@
 ==Seeing the forest in all the trees==
-The following command will give you a hitparade of word-based changes in translation output:
+The following command will give you a hitparade (frequency list) of word-based changes in translation output:
 <pre>
 $ diff -U0 origina_traduko.txt nova_traduko.txt | dwdiff --diff-input |grep -v '^@' |sed 's/.*\[-//'|sed 's/+}.*//'  | sort|uniq -c|sort -n