Difference between revisions of "Apertium-uzb-kaa"
Jump to navigation
Jump to search
m |
|||
| (14 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
==Work plan== |
== Work plan == |
||
* Trimmed coverage means the coverage of both of the morphological analysers after being trimmed according to the bilingual dictionary of the pair, that is, only containing stems which are also in the bilingual dictionary. |
|||
* Testvoc for a category means that the category is [[testvoc]] clean, in both translation directions. |
|||
* Evaluation is taking <math>n</math> words and performing an [[evaluation]] for post-edition word error rate (WER). The output for those <math>n</math> words should be clean. |
|||
* Bidix: number of stems in the bilingual dictionary |
|||
{|class=wikitable |
{|class=wikitable |
||
| Line 11: | Line 6: | ||
! Week |
! Week |
||
! Dates |
! Dates |
||
! |
! Stems goal |
||
! Achieved? |
! Achieved? |
||
! Stems |
|||
! Corpus testvoc uzb->kaa (no *, no */@, no */@/# errors) |
|||
! Cov uzb |
|||
! Corpus testvoc kaa->uzb (no *, no */@, no */@/# errors) |
|||
! Cov kaa |
|||
! Corpus testvoc uzb→kaa <br/> (no *, no */@, no */@/# errors) |
|||
! Corpus testvoc kaa→uzb <br/> (no *, no */@, no */@/# errors) |
|||
! WER,PER uzb→kaa |
|||
! WER,PER kaa→uzb |
|||
!Evaluation |
!Evaluation |
||
!Notes |
!Notes |
||
|- |
|- |
||
| 0 |
|||
| 0 || 20/05—26/05 || 412 || -- || jam(17.7, 17.7, 14.8) udhr(6.1, 6.1, 4.1) bible(9.6, 9.6, 8.2) || jam(34.1, 33.5, 16.2) udhr(n.a. in kaa) bible(17.0, 16.7, 9.7) || -- || |
|||
| 20/05—26/05 |
|||
| -- |
|||
| -- |
|||
| |
|||
* (kaa) 5757 |
|||
* (uzb) 4083 |
|||
* (uzb-kaa) 412 |
|||
| |
|||
* jam(75.22) |
|||
* udhr(34.21) |
|||
* bible(78.80) |
|||
| |
|||
* jam(95.37) |
|||
* bible(68.02) |
|||
| |
|||
* jam(17.7, 17.7, 14.8) |
|||
* udhr(6.1, 6.1, 4.1) |
|||
* bible(9.6, 9.6, 8.2) |
|||
| |
|||
* jam(34.1, 33.5, 16.2) |
|||
* udhr(n.a. in kaa) |
|||
* bible(17.0, 16.7, 9.7) |
|||
| |
|||
* jam(<s>96.19, 93.84</s> TODO properly) |
|||
* bible(TODO) |
|||
| |
|||
* jam(<s>106.15, 102.91</s> TODO properly) |
|||
* bible(TODO) |
|||
| || |
|||
|- |
|- |
||
| 1 |
|||
| 1 || 27/05—02/06 || 5412 || || || || || |
|||
| 27/05—02/06 |
|||
| 5000 from rus-{uzb,kaa}<br/>to each monodix |
|||
| ✘ |
|||
| |
|||
* '''(kaa) 7115''' |
|||
* '''(uzb) 5934''' |
|||
* (uzb-kaa) 412 |
|||
| |
|||
* jam(79.61) |
|||
* udhr(34.21) |
|||
* bible(79.58) |
|||
| |
|||
* jam(95.37) |
|||
* bible(68.18) |
|||
| || || || || || uzb: 6c3887d <br/> kaa: fbb4425 |
|||
|- |
|- |
||
| 2 || 03/06—09/06 || 10412 || || || || || |
|||
|- |
|- |
||
| |
| 2 || 03/06—09/06 || 5000 from rus-{uzb,kaa}<br/>to each monodix || ✓ || || || || || || || || || |
||
|- |
|- |
||
| |
| 3 || 10/06—16/06 || 5000 from rus-{uzb,kaa}<br/>to each monodix || ✓ || || || || || || || || || |
||
|- |
|- |
||
| 4 || 17/06—23/06 || 5000 from rus-{uzb,kaa}<br/>to each monodix || ✓ || || || || || || || || || |
|||
|- |
|||
| 5 |
|||
| 24/06—30/06 |
|||
| Corpus scraping |
|||
| |
|||
| |
|||
* '''(kaa) 25513''' |
|||
* '''(uzb) 34358''' |
|||
* (uzb-kaa) 412 |
|||
| |
|||
* jam(85.31) |
|||
* udhr(39.47) |
|||
* bible(92.63) |
|||
| |
|||
* jam(96.57) |
|||
* bible(93.01) |
|||
| || || || || || uzb: 432af5d <br/> kaa: 7a02618 <br/> uzb-kaa: c5ea3c3 |
|||
|- |
|||
| 13 |
|||
| 19/08-25/08 |
|||
| Bidix |
|||
| |
|||
| |
|||
* (kaa) 25512 |
|||
* (uzb) 34370 |
|||
* '''(uzb-kaa) 19363''' |
|||
| |
|||
* jam(81.80) |
|||
* udhr(39.47) |
|||
* bible(94.64) |
|||
| |
|||
* jam(96.57) |
|||
* bible(93.18) |
|||
| |
|||
* jam(49.7, 49.7, 34.1) |
|||
* udhr(62.0, 60.1, 36.0) |
|||
* bible(49.1, 48.4, 30.3) |
|||
| |
|||
* jam(72.9, 71.4, 45.2) |
|||
* udhr(n.a. in kaa) |
|||
* bible(56.0, 53.9, 38.8) |
|||
| |
|||
* jam(69.74, 65.99) |
|||
* bible(112.11, 97.13) |
|||
| |
|||
* jam(87.58, 84.16) |
|||
* bible(121.10, 104.57) |
|||
| || uzb: 4c121f4 kaa: 02f120d uzb-kaa: 25d79f6 |
|||
|} |
|||
<!-- |
<!-- |
||
| Line 46: | Line 140: | ||
| 12 || 12/08—18/08 || || || || || |
| 12 || 12/08—18/08 || || || || || |
||
|- |
|- |
||
|} |
|||
--> |
--> |
||
== Comments == |
|||
* Corpus testvoc := cat <txt> | apertium-uzb-kaa/testvoc/corpus/trimmed-coverage.sh {uzb-kaa,kaa-uzb}. Corpora can be found [https://github.com/taruen/apertiumpp/tree/master/data4apertium/corpora here]. |
|||
* Number of stems taken from the header "aperium-dixtools format-1line 10 55 apertium-uzb-kaa.uzb-kaa.dix" produces. |
|||
* Bidix: number of stems in the bilingual dictionary. |
|||
** Number of stems is taken from the header "aperium-dixtools format-1line 10 55 apertium-uzb-kaa.uzb-kaa.dix" produces. |
|||
* Corpus testvoc: cat <txt> | apertium-uzb-kaa/testvoc/corpus/trimmed-coverage.sh {uzb-kaa,kaa-uzb}. Corpora can be found [https://github.com/taruen/apertiumpp/tree/master/data4apertium/corpora here]. |
|||
* WER is calculated like shown below. Currently WER numbers are only a crude approximation, since input text and -ref text are probably independent translations (not parallel). |
|||
* Evaluation is taking <math>n</math> words and performing an [[evaluation]] for post-edition word error rate (WER). The output for those <math>n</math> words should be clean. |
|||
Related terms: |
|||
* Trimmed coverage means the coverage of both of the morphological analysers after being trimmed according to the bilingual dictionary of the pair, that is, only containing stems which are also in the bilingual dictionary. |
|||
* Testvoc for a category means that the category is [[testvoc]] clean, in both translation directions. |
|||
<pre> |
|||
apertium-uzb-kaa$ perl ../../../../../../src/sourceforge-apertium/trunk/apertium-eval-translator/apertium-eval-translator.pl -test <(cat ../../../data4apertium/corpora/jam/uzb.txt | apertium -d . uzb-kaa) -ref ../../../data4apertium/corpora/jam/kaa.txt |
|||
Test file: '/dev/fd/63' |
|||
Reference file '../../../data4apertium/corpora/jam/kaa.txt' |
|||
Statistics about input files |
|||
------------------------------------------------------- |
|||
Number of words in reference: 341 |
|||
Number of words in test: 309 |
|||
Number of unknown words (marked with a star) in test: 284 |
|||
Percentage of unknown words: 91.91 % |
|||
Results when removing unknown-word marks (stars) |
|||
------------------------------------------------------- |
|||
Edit distance: 325 |
|||
Word error rate (WER): 95.31 % |
|||
Number of position-independent correct words: 26 |
|||
Position-independent word error rate (PER): 92.38 % |
|||
Results when unknown-word marks (stars) are not removed |
|||
------------------------------------------------------- |
|||
Edit distance: 328 |
|||
Word Error Rate (WER): 96.19 % |
|||
Number of position-independent correct words: 21 |
|||
Position-independent word error rate (PER): 93.84 % |
|||
Statistics about the translation of unknown words |
|||
------------------------------------------------------- |
|||
Number of unknown words which were free rides: 3 |
|||
Percentage of unknown words that were free rides: 1.06 % |
|||
</pre> |
|||
Latest revision as of 14:34, 1 September 2019
Work plan[edit]
| Week | Dates | Stems goal | Achieved? | Stems | Cov uzb | Cov kaa | Corpus testvoc uzb→kaa (no *, no */@, no */@/# errors) |
Corpus testvoc kaa→uzb (no *, no */@, no */@/# errors) |
WER,PER uzb→kaa | WER,PER kaa→uzb | Evaluation | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20/05—26/05 | -- | -- |
|
|
|
|
|
|
|
||
| 1 | 27/05—02/06 | 5000 from rus-{uzb,kaa} to each monodix |
✘ |
|
|
|
uzb: 6c3887d kaa: fbb4425 | |||||
| 2 | 03/06—09/06 | 5000 from rus-{uzb,kaa} to each monodix |
✓ | |||||||||
| 3 | 10/06—16/06 | 5000 from rus-{uzb,kaa} to each monodix |
✓ | |||||||||
| 4 | 17/06—23/06 | 5000 from rus-{uzb,kaa} to each monodix |
✓ | |||||||||
| 5 | 24/06—30/06 | Corpus scraping |
|
|
|
uzb: 432af5d kaa: 7a02618 uzb-kaa: c5ea3c3 | ||||||
| 13 | 19/08-25/08 | Bidix |
|
|
|
|
|
|
|
uzb: 4c121f4 kaa: 02f120d uzb-kaa: 25d79f6 |
Comments[edit]
- Bidix: number of stems in the bilingual dictionary.
- Number of stems is taken from the header "aperium-dixtools format-1line 10 55 apertium-uzb-kaa.uzb-kaa.dix" produces.
- Corpus testvoc: cat <txt> | apertium-uzb-kaa/testvoc/corpus/trimmed-coverage.sh {uzb-kaa,kaa-uzb}. Corpora can be found here.
- WER is calculated like shown below. Currently WER numbers are only a crude approximation, since input text and -ref text are probably independent translations (not parallel).
- Evaluation is taking words and performing an evaluation for post-edition word error rate (WER). The output for those words should be clean.
Related terms:
- Trimmed coverage means the coverage of both of the morphological analysers after being trimmed according to the bilingual dictionary of the pair, that is, only containing stems which are also in the bilingual dictionary.
- Testvoc for a category means that the category is testvoc clean, in both translation directions.
apertium-uzb-kaa$ perl ../../../../../../src/sourceforge-apertium/trunk/apertium-eval-translator/apertium-eval-translator.pl -test <(cat ../../../data4apertium/corpora/jam/uzb.txt | apertium -d . uzb-kaa) -ref ../../../data4apertium/corpora/jam/kaa.txt Test file: '/dev/fd/63' Reference file '../../../data4apertium/corpora/jam/kaa.txt' Statistics about input files ------------------------------------------------------- Number of words in reference: 341 Number of words in test: 309 Number of unknown words (marked with a star) in test: 284 Percentage of unknown words: 91.91 % Results when removing unknown-word marks (stars) ------------------------------------------------------- Edit distance: 325 Word error rate (WER): 95.31 % Number of position-independent correct words: 26 Position-independent word error rate (PER): 92.38 % Results when unknown-word marks (stars) are not removed ------------------------------------------------------- Edit distance: 328 Word Error Rate (WER): 96.19 % Number of position-independent correct words: 21 Position-independent word error rate (PER): 93.84 % Statistics about the translation of unknown words ------------------------------------------------------- Number of unknown words which were free rides: 3 Percentage of unknown words that were free rides: 1.06 %
