Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on
If you have any questions, please come and talk to us on
#apertium
on irc.freenode.net
or contact the GitHub migration team.Comparison of part-of-speech tagging systems
From Apertium
(Difference between revisions)
Line 10: | Line 10: | ||
! Catalan !! Spanish !! Serbo-Croatian !! Russian !! Kazakh !! Portuguese !! Swedish |
! Catalan !! Spanish !! Serbo-Croatian !! Russian !! Kazakh !! Portuguese !! Swedish |
||
|- |
|- |
||
− | ! <small>23,673</small> !! <small>20,487</small> !! <small>20,128</small> !! <small>10,171</small> !! <small>4,348</small> !! <small>3,823 </small> !! <small>239</small> |
+ | ! <small>23,673</small> !! <small>20,487</small> !! <small>20,128</small> !! <small>10,171</small> !! <small>4,348</small> !! <small>5,718</small> !! <small>239</small> |
|- |
|- |
||
− | | '''1st''' ||align=right| 81.66 ||align=right| 86.23 ||align=right| 75.22 ||align=right| 75.63 ||align=right| 80.79||align=right| 61.53 ||align=right| 72.90 |
+ | | '''1st''' ||align=right| 81.66 ||align=right| 86.23 ||align=right| 75.22 ||align=right| 75.63 ||align=right| 80.79||align=right| 66.58 |||align=right| 72.90 |
|- |
|- |
||
− | | '''CG→1st''' ||align=right| 83.79 ||align=right| 87.35 ||align=right| 79.67 ||align=right| 79.52 ||align=right| 86.19 ||align=right| 63.33 ||align=right| 73.86 |
+ | | '''CG→1st''' ||align=right| 83.79 ||align=right| 87.35 ||align=right| 79.67 ||align=right| 79.52 ||align=right| 86.19 ||align=right| 77.51 |||align=right| 73.86 |
|- |
|- |
||
− | | '''Unigram model 1''' ||align=right| 91.72±1.37 ||align=right| 91.41±1.31 ||||||||align=right| 63.03±3.27 |
+ | | '''Unigram model 1''' ||align=right| 91.72±1.37 ||align=right| 91.41±1.31 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Unigram model 1''' ||align=right| 92.37±1.33 ||align=right| 92.52±1.18 ||||||||align=right| 63.29±3.24 |
+ | | '''CG→Unigram model 1''' ||align=right| 92.37±1.33 ||align=right| 92.52±1.18 ||||||||align=right| |
|- |
|- |
||
− | | '''Unigram model 2''' ||align=right| 91.78±1.30 ||align=right| 91.03±1.25 ||||||||align=right| 63.23±3.41 |
+ | | '''Unigram model 2''' ||align=right| 91.78±1.30 ||align=right| 91.03±1.25 ||||||||align=right| 77.35±5.20 |
|- |
|- |
||
− | | '''CG→Unigram model 2''' ||align=right| 92.06±1.30 ||align=right| 91.94±1.10 ||||||||align=right| 63.16±3.17 |
+ | | '''CG→Unigram model 2''' ||align=right| 92.06±1.30 ||align=right| 91.94±1.10 ||||||||align=right| 79.19±5.66 |
|- |
|- |
||
− | | '''Unigram model 3''' ||align=right| 91.74±1.29 ||align=right| 91.01±1.25 ||||||||align=right| 63.23±3.41 |
+ | | '''Unigram model 3''' ||align=right| 91.74±1.29 ||align=right| 91.01±1.25 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Unigram model 3''' ||align=right| 92.03±1.29 ||align=right| 91.91±1.08 ||||||||align=right| 63.16±3.17 |
+ | | '''CG→Unigram model 3''' ||align=right| 92.03±1.29 ||align=right| 91.91±1.08 ||||||||align=right| 79.19±5.66 |
|- |
|- |
||
− | | '''Bigram (unsup, 0 iters)''' ||align=right| 85.05±1.22 ||align=right| 83.60±1.94 ||||||||align=right| 62.99±3.11 |
+ | | '''Bigram (unsup, 0 iters)''' ||align=right| 85.05±1.22 ||align=right| 83.60±1.94 ||||||||align=right| 71.28±3.75 |
|- |
|- |
||
− | | '''Bigram (unsup, 50 iters)''' ||align=right| 88.81±1.36 ||align=right| 87.37±2.03 ||||||||align=right| 61.31±3.43 |
+ | | '''Bigram (unsup, 50 iters)''' ||align=right| 88.81±1.36 ||align=right| 87.37±2.03 ||||||||align=right| |
|- |
|- |
||
− | | '''Bigram (unsup, 250 iters)''' ||align=right| 88.53±1.35 ||align=right| 86.99±2.03 ||||||||align=right| 61.21±3.50 |
+ | | '''Bigram (unsup, 250 iters)''' ||align=right| 88.53±1.35 ||align=right| 86.99±2.03 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Bigram (unsup, 0 iters)''' ||align=right| 88.96±1.21 ||align=right| 87.76±1.82 ||||||||align=right| 63.01±3.23 |
+ | | '''CG→Bigram (unsup, 0 iters)''' ||align=right| 88.96±1.21 ||align=right| 87.76±1.82 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Bigram (unsup, 50 iters)''' ||align=right| 90.77±1.68 ||align=right| 89.34±1.71 ||||||||align=right| 62.82±3.26 |
+ | | '''CG→Bigram (unsup, 50 iters)''' ||align=right| 90.77±1.68 ||align=right| 89.34±1.71 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Bigram (unsup, 250 iters)''' ||align=right| 90.54±1.67 ||align=right| 89.33±1.71 ||||||||align=right| 62.82±3.26 |
+ | | '''CG→Bigram (unsup, 250 iters)''' ||align=right| 90.54±1.67 ||align=right| 89.33±1.71 ||||||||align=right| |
|- |
|- |
||
− | | '''Bigram (sup)''' ||align=right| 94.60±1.06 ||align=right| 93.52±1.46 ||||||||align=right| 63.14±3.24 |
+ | | '''Bigram (sup)''' ||align=right| 94.60±1.06 ||align=right| 93.52±1.46 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Bigram (sup)''' ||align=right| 94.62±1.38 ||align=right| 92.70±1.60 ||||||||align=right| 63.09±3.37 |
+ | | '''CG→Bigram (sup)''' ||align=right| 94.62±1.38 ||align=right| 92.70±1.60 ||||||||align=right| |
|- |
|- |
||
− | | '''Lwsw (0 iters)''' ||align=right| 90.16±1.00 ||align=right| 89.78±1.27 ||||||||align=right| 62.80±3.67 |
+ | | '''Lwsw (0 iters)''' ||align=right| 90.16±1.00 ||align=right| 89.78±1.27 ||||||||align=right| 73.13±3.87 |
|- |
|- |
||
− | | '''Lwsw (50 iters)''' ||align=right| 90.51±0.98 ||align=right| 89.98±1.38 ||||||||align=right| 62.74±3.62 |
+ | | '''Lwsw (50 iters)''' ||align=right| 90.51±0.98 ||align=right| 89.98±1.38 ||||||||align=right| 72.90±3.97 |
|- |
|- |
||
− | | '''Lwsw (250 iters)''' ||align=right| 90.51±0.98 ||align=right| 90.06±1.39 ||||||||align=right| 62.74±3.62 |
+ | | '''Lwsw (250 iters)''' ||align=right| 90.51±0.98 ||align=right| 90.06±1.39 ||||||||align=right| 72.87±4.09 |
|- |
|- |
||
− | | '''CG→Lwsw (0 iters)''' ||align=right| 90.78±1.26 ||align=right| 89.61±1.43 ||||||||align=right| 62.73±3.55 |
+ | | '''CG→Lwsw (0 iters)''' ||align=right| 90.78±1.26 ||align=right| 89.61±1.43 ||||||||align=right| 77.20±5.11 |
|- |
|- |
||
− | | '''CG→Lwsw (50 iters)''' ||align=right| 91.05±1.21 ||align=right| 89.63±1.56 ||||||||align=right| 62.73±3.55 |
+ | | '''CG→Lwsw (50 iters)''' ||align=right| 91.05±1.21 ||align=right| 89.63±1.56 ||||||||align=right| |
|- |
|- |
||
− | | '''CG→Lwsw (250 iters)''' ||align=right| 91.06±1.25 ||align=right| 89.67±1.58 ||||||||align=right| 62.73±3.55 |
+ | | '''CG→Lwsw (250 iters)''' ||align=right| 91.06±1.25 ||align=right| 89.67±1.58 ||||||||align=right| |
|- |
|- |
||
| '''kaz-tagger''' || |
| '''kaz-tagger''' || |
Revision as of 14:16, 31 May 2016
|
Apertium would like to have really good part-of-speech tagging, but in many cases falls below the state-of-the-art (around 97% tagging accuracy). This page intends to collect a comparison of tagging systems in Apertium and give some ideas of what could be done to improve them.
In the following table values of the form x±y are the sample mean and standard deviation of the results of 10-fold cross validation.
System | Language | ||||||
---|---|---|---|---|---|---|---|
Catalan | Spanish | Serbo-Croatian | Russian | Kazakh | Portuguese | Swedish | |
23,673 | 20,487 | 20,128 | 10,171 | 4,348 | 5,718 | 239 | |
1st | 81.66 | 86.23 | 75.22 | 75.63 | 80.79 | 66.58 | align=right| 72.90 |
CG→1st | 83.79 | 87.35 | 79.67 | 79.52 | 86.19 | 77.51 | align=right| 73.86 |
Unigram model 1 | 91.72±1.37 | 91.41±1.31 | |||||
CG→Unigram model 1 | 92.37±1.33 | 92.52±1.18 | |||||
Unigram model 2 | 91.78±1.30 | 91.03±1.25 | 77.35±5.20 | ||||
CG→Unigram model 2 | 92.06±1.30 | 91.94±1.10 | 79.19±5.66 | ||||
Unigram model 3 | 91.74±1.29 | 91.01±1.25 | |||||
CG→Unigram model 3 | 92.03±1.29 | 91.91±1.08 | 79.19±5.66 | ||||
Bigram (unsup, 0 iters) | 85.05±1.22 | 83.60±1.94 | 71.28±3.75 | ||||
Bigram (unsup, 50 iters) | 88.81±1.36 | 87.37±2.03 | |||||
Bigram (unsup, 250 iters) | 88.53±1.35 | 86.99±2.03 | |||||
CG→Bigram (unsup, 0 iters) | 88.96±1.21 | 87.76±1.82 | |||||
CG→Bigram (unsup, 50 iters) | 90.77±1.68 | 89.34±1.71 | |||||
CG→Bigram (unsup, 250 iters) | 90.54±1.67 | 89.33±1.71 | |||||
Bigram (sup) | 94.60±1.06 | 93.52±1.46 | |||||
CG→Bigram (sup) | 94.62±1.38 | 92.70±1.60 | |||||
Lwsw (0 iters) | 90.16±1.00 | 89.78±1.27 | 73.13±3.87 | ||||
Lwsw (50 iters) | 90.51±0.98 | 89.98±1.38 | 72.90±3.97 | ||||
Lwsw (250 iters) | 90.51±0.98 | 90.06±1.39 | 72.87±4.09 | ||||
CG→Lwsw (0 iters) | 90.78±1.26 | 89.61±1.43 | 77.20±5.11 | ||||
CG→Lwsw (50 iters) | 91.05±1.21 | 89.63±1.56 | |||||
CG→Lwsw (250 iters) | 91.06±1.25 | 89.67±1.58 | |||||
kaz-tagger | |||||||
CG→kaz-tagger |
In the following table, the intervals represent the [low, high] values from 10-fold cross validation.
Language | Corpus | System | |||||||
---|---|---|---|---|---|---|---|---|---|
Sent | Tok | Amb | 1st | CG+1st | Unigram | CG+Unigram | apertium-tagger | CG+apertium-tagger | |
Catalan | 1,413 | 24,144 | ? | 81.85 | 83.96 | [75.65, 78.46] | [87.76, 90.48] | [94.16, 96.28] | [93.92, 96.16] |
Spanish | 1,271 | 21,247 | ? | 86.18 | 86.71 | [78.20, 80.06] | [87.72, 90.27] | [90.15, 94.86] | [91.84, 93.70] |
Serbo-Croatian | 1,190 | 20,128 | ? | 75.22 | 79.67 | [75.36, 78.79] | [75.36, 77.28] | ||
Russian | 451 | 10,171 | ? | 75.63 | 79.52 | [70.49, 72.94] | [74.68, 78.65] | n/a | n/a |
Kazakh | 403 | 4,348 | ? | 80.79 | 86.19 | [84.36, 87.79] | [85.56, 88.72] | n/a | n/a |
Portuguese | 119 | 3,823 | ? | 72.54 | 87.34 | [77.10, 87.72] | [84.05, 91.96] | ||
Swedish | 11 | 239 | ? | 72.90 | 73.86 | [56.00, 82.97] |
Sent = sentences, Tok = tokens, Amb = average ambiguity from the morphological analyser
Systems
-
1st
: Selects the first analysis from the morphological analyser -
CG
: Uses the CG (from the monolingual language package in languages) to preprocess the input. -
Unigram
: Lexicalised unigram tagger -
apertium-tagger
: Uses the bigram HMM tagger included with Apertium.
Corpora
The tagged corpora used in the experiments are found in the monolingual packages in languages, under the texts/
subdirectory.
Todo
- Implement this tagger: https://spacy.io/blog/part-of-speech-POS-tagger-in-python