Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Comparison of part-of-speech tagging systems

From Apertium
(Difference between revisions)
Jump to: navigation, search
(Undo revision 58286 by Frankier (talk))
Line 10: Line 10:
 
! Catalan !! Spanish !! Serbo-Croatian !! Russian !! Kazakh !! Portuguese !! Swedish
  
! Catalan !! Spanish !! Serbo-Croatian !! Russian !! Kazakh !! Portuguese !! Swedish
 
|-
  
|-
−
! <small>23,673</small> !! <small>20,487</small> !! <small>20,128</small> !! <small>10,171</small> !! <small>4,348</small> !! <small>3,823 </small> !! <small>239</small>
 +
! <small>23,673</small> !! <small>20,487</small> !! <small>20,128</small> !! <small>10,171</small> !! <small>4,348</small> !! <small>5,718</small> !! <small>239</small>
 
|-
  
|-
−
| '''1st''' ||align=right| 81.66 ||align=right| 86.23 ||align=right| 75.22 ||align=right| 75.63 ||align=right| 80.79||align=right| 61.53 ||align=right| 72.90
 +
| '''1st''' ||align=right| 81.66 ||align=right| 86.23 ||align=right| 75.22 ||align=right| 75.63 ||align=right| 80.79||align=right| 66.58 |||align=right| 72.90
 
|-
  
|-
−
| '''CG→1st''' ||align=right| 83.79 ||align=right| 87.35 ||align=right| 79.67 ||align=right| 79.52 ||align=right| 86.19 ||align=right| 63.33 ||align=right| 73.86
 +
| '''CG→1st''' ||align=right| 83.79 ||align=right| 87.35 ||align=right| 79.67 ||align=right| 79.52 ||align=right| 86.19 ||align=right| 77.51 |||align=right| 73.86
 
|-
  
|-
−
| '''Unigram model 1''' ||align=right| 91.72±1.37 ||align=right| 91.41±1.31 ||||||||align=right| 63.03±3.27
+
| '''Unigram model 1''' ||align=right| 91.72±1.37 ||align=right| 91.41±1.31 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Unigram model 1''' ||align=right| 92.37±1.33 ||align=right| 92.52±1.18 ||||||||align=right| 63.29±3.24
+
| '''CG→Unigram model 1''' ||align=right| 92.37±1.33 ||align=right| 92.52±1.18 ||||||||align=right|
 
|-
  
|-
−
| '''Unigram model 2''' ||align=right| 91.78±1.30 ||align=right| 91.03±1.25 ||||||||align=right| 63.23±3.41
+
| '''Unigram model 2''' ||align=right| 91.78±1.30 ||align=right| 91.03±1.25 ||||||||align=right| 77.35±5.20
 
|-
  
|-
−
| '''CG→Unigram model 2''' ||align=right| 92.06±1.30 ||align=right| 91.94±1.10 ||||||||align=right| 63.16±3.17
+
| '''CG→Unigram model 2''' ||align=right| 92.06±1.30 ||align=right| 91.94±1.10 ||||||||align=right| 79.19±5.66
 
|-
  
|-
−
| '''Unigram model 3''' ||align=right| 91.74±1.29 ||align=right| 91.01±1.25 ||||||||align=right| 63.23±3.41
+
| '''Unigram model 3''' ||align=right| 91.74±1.29 ||align=right| 91.01±1.25 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Unigram model 3''' ||align=right| 92.03±1.29 ||align=right| 91.91±1.08 ||||||||align=right| 63.16±3.17
+
| '''CG→Unigram model 3''' ||align=right| 92.03±1.29 ||align=right| 91.91±1.08 ||||||||align=right| 79.19±5.66
 
|-
  
|-
−
| '''Bigram (unsup, 0 iters)''' ||align=right| 85.05±1.22 ||align=right| 83.60±1.94 ||||||||align=right| 62.99±3.11
+
| '''Bigram (unsup, 0 iters)''' ||align=right| 85.05±1.22 ||align=right| 83.60±1.94 ||||||||align=right| 71.28±3.75
 
|-
  
|-
−
| '''Bigram (unsup, 50 iters)''' ||align=right| 88.81±1.36 ||align=right| 87.37±2.03 ||||||||align=right| 61.31±3.43
+
| '''Bigram (unsup, 50 iters)''' ||align=right| 88.81±1.36 ||align=right| 87.37±2.03 ||||||||align=right|
 
|-
  
|-
−
| '''Bigram (unsup, 250 iters)''' ||align=right| 88.53±1.35 ||align=right| 86.99±2.03 ||||||||align=right| 61.21±3.50
+
| '''Bigram (unsup, 250 iters)''' ||align=right| 88.53±1.35 ||align=right| 86.99±2.03 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Bigram (unsup, 0 iters)''' ||align=right| 88.96±1.21 ||align=right| 87.76±1.82 ||||||||align=right| 63.01±3.23
+
| '''CG→Bigram (unsup, 0 iters)''' ||align=right| 88.96±1.21 ||align=right| 87.76±1.82 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Bigram (unsup, 50 iters)''' ||align=right| 90.77±1.68 ||align=right| 89.34±1.71 ||||||||align=right| 62.82±3.26
+
| '''CG→Bigram (unsup, 50 iters)''' ||align=right| 90.77±1.68 ||align=right| 89.34±1.71 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Bigram (unsup, 250 iters)''' ||align=right| 90.54±1.67 ||align=right| 89.33±1.71 ||||||||align=right| 62.82±3.26
+
| '''CG→Bigram (unsup, 250 iters)''' ||align=right| 90.54±1.67 ||align=right| 89.33±1.71 ||||||||align=right|
 
|-
  
|-
−
| '''Bigram (sup)''' ||align=right| 94.60±1.06 ||align=right| 93.52±1.46 ||||||||align=right| 63.14±3.24
+
| '''Bigram (sup)''' ||align=right| 94.60±1.06 ||align=right| 93.52±1.46 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Bigram (sup)''' ||align=right| 94.62±1.38 ||align=right| 92.70±1.60 ||||||||align=right| 63.09±3.37
+
| '''CG→Bigram (sup)''' ||align=right| 94.62±1.38 ||align=right| 92.70±1.60 ||||||||align=right|
 
|-
  
|-
−
| '''Lwsw (0 iters)''' ||align=right| 90.16±1.00 ||align=right| 89.78±1.27 ||||||||align=right| 62.80±3.67
+
| '''Lwsw (0 iters)''' ||align=right| 90.16±1.00 ||align=right| 89.78±1.27 ||||||||align=right| 73.13±3.87
 
|-
  
|-
−
| '''Lwsw (50 iters)''' ||align=right| 90.51±0.98 ||align=right| 89.98±1.38 ||||||||align=right| 62.74±3.62
+
| '''Lwsw (50 iters)''' ||align=right| 90.51±0.98 ||align=right| 89.98±1.38 ||||||||align=right| 72.90±3.97
 
|-
  
|-
−
| '''Lwsw (250 iters)''' ||align=right| 90.51±0.98 ||align=right| 90.06±1.39 ||||||||align=right| 62.74±3.62
+
| '''Lwsw (250 iters)''' ||align=right| 90.51±0.98 ||align=right| 90.06±1.39 ||||||||align=right| 72.87±4.09
 
|-
  
|-
−
| '''CG→Lwsw (0 iters)''' ||align=right| 90.78±1.26 ||align=right| 89.61±1.43 ||||||||align=right| 62.73±3.55
+
| '''CG→Lwsw (0 iters)''' ||align=right| 90.78±1.26 ||align=right| 89.61±1.43 ||||||||align=right| 77.20±5.11
 
|-
  
|-
−
| '''CG→Lwsw (50 iters)''' ||align=right| 91.05±1.21 ||align=right| 89.63±1.56 ||||||||align=right| 62.73±3.55
+
| '''CG→Lwsw (50 iters)''' ||align=right| 91.05±1.21 ||align=right| 89.63±1.56 ||||||||align=right|
 
|-
  
|-
−
| '''CG→Lwsw (250 iters)''' ||align=right| 91.06±1.25 ||align=right| 89.67±1.58 ||||||||align=right| 62.73±3.55
+
| '''CG→Lwsw (250 iters)''' ||align=right| 91.06±1.25 ||align=right| 89.67±1.58 ||||||||align=right|
 
|-
  
|-
 
| '''kaz-tagger''' ||
  
| '''kaz-tagger''' ||

Revision as of 14:16, 31 May 2016

Contents

Apertium would like to have really good part-of-speech tagging, but in many cases falls below the state-of-the-art (around 97% tagging accuracy). This page intends to collect a comparison of tagging systems in Apertium and give some ideas of what could be done to improve them.

In the following table values of the form x±y are the sample mean and standard deviation of the results of 10-fold cross validation.

System Language
Catalan Spanish Serbo-Croatian Russian Kazakh Portuguese Swedish
23,673 20,487 20,128 10,171 4,348 5,718 239
1st 81.66 86.23 75.22 75.63 80.79 66.58 align=right| 72.90
CG→1st 83.79 87.35 79.67 79.52 86.19 77.51 align=right| 73.86
Unigram model 1 91.72±1.37 91.41±1.31
CG→Unigram model 1 92.37±1.33 92.52±1.18
Unigram model 2 91.78±1.30 91.03±1.25 77.35±5.20
CG→Unigram model 2 92.06±1.30 91.94±1.10 79.19±5.66
Unigram model 3 91.74±1.29 91.01±1.25
CG→Unigram model 3 92.03±1.29 91.91±1.08 79.19±5.66
Bigram (unsup, 0 iters) 85.05±1.22 83.60±1.94 71.28±3.75
Bigram (unsup, 50 iters) 88.81±1.36 87.37±2.03
Bigram (unsup, 250 iters) 88.53±1.35 86.99±2.03
CG→Bigram (unsup, 0 iters) 88.96±1.21 87.76±1.82
CG→Bigram (unsup, 50 iters) 90.77±1.68 89.34±1.71
CG→Bigram (unsup, 250 iters) 90.54±1.67 89.33±1.71
Bigram (sup) 94.60±1.06 93.52±1.46
CG→Bigram (sup) 94.62±1.38 92.70±1.60
Lwsw (0 iters) 90.16±1.00 89.78±1.27 73.13±3.87
Lwsw (50 iters) 90.51±0.98 89.98±1.38 72.90±3.97
Lwsw (250 iters) 90.51±0.98 90.06±1.39 72.87±4.09
CG→Lwsw (0 iters) 90.78±1.26 89.61±1.43 77.20±5.11
CG→Lwsw (50 iters) 91.05±1.21 89.63±1.56
CG→Lwsw (250 iters) 91.06±1.25 89.67±1.58
kaz-tagger
CG→kaz-tagger

In the following table, the intervals represent the [low, high] values from 10-fold cross validation.

Language Corpus System
Sent Tok Amb 1st CG+1st Unigram CG+Unigram apertium-tagger CG+apertium-tagger
Catalan 1,413 24,144  ? 81.85 83.96 [75.65, 78.46] [87.76, 90.48] [94.16, 96.28] [93.92, 96.16]
Spanish 1,271 21,247  ? 86.18 86.71 [78.20, 80.06] [87.72, 90.27] [90.15, 94.86] [91.84, 93.70]
Serbo-Croatian 1,190 20,128  ? 75.22 79.67 [75.36, 78.79] [75.36, 77.28]
Russian 451 10,171  ? 75.63 79.52 [70.49, 72.94] [74.68, 78.65] n/a n/a
Kazakh 403 4,348  ? 80.79 86.19 [84.36, 87.79] [85.56, 88.72] n/a n/a
Portuguese 119 3,823  ? 72.54 87.34 [77.10, 87.72] [84.05, 91.96]
Swedish 11 239  ? 72.90 73.86 [56.00, 82.97]

Sent = sentences, Tok = tokens, Amb = average ambiguity from the morphological analyser

Systems

  • 1st: Selects the first analysis from the morphological analyser
  • CG: Uses the CG (from the monolingual language package in languages) to preprocess the input.
  • Unigram: Lexicalised unigram tagger
  • apertium-tagger: Uses the bigram HMM tagger included with Apertium.

Corpora

The tagged corpora used in the experiments are found in the monolingual packages in languages, under the texts/ subdirectory.

Todo

Retrieved from "http://wiki.apertium.org/w/index.php?title=Comparison_of_part-of-speech_tagging_systems&oldid=58297"
Personal tools