Translation quality statistics - Revision history

Unhammer: /* Quality Evaluations */

2025-11-25T17:41:18Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2025-11-25T17:40:48Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2024-06-03T10:23:54Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2024-06-03T10:23:39Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2024-06-03T10:22:16Z

Quality Evaluations

Unhammer at 09:10, 21 November 2021

2021-11-21T09:10:06Z

Unhammer: /* Quality Evaluations */

2021-11-11T10:04:04Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2021-11-11T09:56:09Z

Quality Evaluations

Unhammer: /* Quality Evaluations */

2021-11-11T09:28:34Z

Quality Evaluations

Sushain: /* Coverage and Dictionary size */

2018-04-26T19:52:51Z

Coverage and Dictionary size

@@ Line 26: / Line 26: @@
 |  bg → mk || - || - ||
 |-
-|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || 0.898 ||rowspan=2| WER from Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011. '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering). BLEU results are from from [https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/3157630]
+|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || 0.898 ||rowspan=2| WER from Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011. '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering). BLEU results are from [https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/3157630]
 |-
 |  nob → nno ||32.5%, 17.7% || - || 0.817

@@ Line 26: / Line 26: @@
 |  bg → mk || - || - ||
 |-
-|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011. '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering)
+|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || 0.898 ||rowspan=2| WER from Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011. '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering). BLEU results are from from [https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/3157630]
 |-
-|  nob → nno ||32.5%, 17.7% || - || 0.74
+|  nob → nno ||32.5%, 17.7% || - || 0.817
 |-
 |rowspan=2| <code>[[apertium-br-fr]]</code> ||rowspan=2| March&nbsp;2010 ||rowspan=2| 0.2.0 || br → fr ||rowspan=2 {{no}} ||  38 % || 22 % || - ||rowspan=2| Tyers, 2010

@@ Line 26: / Line 26: @@
 |  bg → mk || - || - ||
 |-
-|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011). '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering)
+|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011. '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering)
 |-
 |  nob → nno ||32.5%, 17.7% || - || 0.74

@@ Line 26: / Line 26: @@
 |  bg → mk || - || - ||
 |-
-|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations; as of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011). '''Update''': As of June 2024, apertium-nno-nob from git used to translate news articles had WER below 0.6 % (so on average, every 166th word needed altering)
+|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations). '''2021 Update''': As of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011). '''2024 Update''': As of June 2024, apertium-nno-nob from git used to translate fresh news articles had WER below 0.6 % (so on average, every 166th word needed altering)
 |-
 |  nob → nno ||32.5%, 17.7% || - || 0.74

@@ Line 26: / Line 26: @@
 |  bg → mk || - || - ||
 |-
-|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations; as of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011)
+|rowspan=2| <code>[[apertium-nno-nob]]</code> ||rowspan=2|12th&nbsp;October&nbsp;2009||rowspan=2| 0.6.1  || nno → nob ||rowspan=2 {{yes}} ||  - || - || - ||rowspan=2| Unhammer and Trosterud, 2009<br/> (two reference translations; as of [https://github.com/apertium/apertium-nno-nob/commit/02ea2f196848e323b0898e9225334182968e7894 Nov 2021] the [https://github.com/apertium/apertium-nno-nob/blob/master/freerbmt09_nnnb/eval/500.apertium.testset same test set] gives 0.862±0.011). '''Update''': As of June 2024, apertium-nno-nob from git used to translate news articles had WER below 0.6 % (so on average, every 166th word needed altering)
 |-
 |  nob → nno ||32.5%, 17.7% || - || 0.74

@@ Line 1: / Line 1: @@
 This page aims to give an overview of the ''quality'' of various translators available in the Apertium platform.
+The metrics used here don't tell the whole story – see [[Evaluation]] for some discussion of strengths and limitations of WER/BLEU.
 ==Quality Evaluations==

@@ Line 72: / Line 72: @@
 The number of entries in a dictionary, as well as the number of corpus forms that get some analysis, may give an indication of the maturity of a language pair.
-For most dictionaries, there are numbers at least for dictionary size on the wiki, some also have coverage stats, see [[:Category:Datastats]]. The stats of a certain apertium package will have a page with that package name (the name of the directory in SVN) followed by "/stats", e.g. [[apertium-es-ca/stats]]. Some language pairs are split into several packages, so for nno-nob, there are pages [[apertium-nno]], [[apertium-nob]] and [[apertium-nno-nob]], but for dictionary counts you should consult the last one.
+For most dictionaries, there are numbers at least for dictionary size on the wiki, some also have coverage stats, see [[:Category:Datastats]]. The stats of a certain apertium package will have a page with that package name (the name of the repository in GitHub) followed by "/stats", e.g. [[apertium-es-ca/stats]]. Some language pairs are split into several packages, so for nno-nob, there are pages [[apertium-nno]], [[apertium-nob]] and [[apertium-nno-nob]], but for dictionary counts you should consult the last one.
 (Stats pages currently do not show number of CG or transfer rules.)