Talk:Scandinavian MT project

From Apertium
Revision as of 09:03, 8 March 2016 by Unhammer (talk | contribs)
Jump to navigation Jump to search

Coverage on Wikipedia dumps ("w/o cmp" is with decompounding turned off, ie. without the -e switch to lt-proc).

E.g.

bzcat ~/corpora/nnclean2.txt.bz2 \
  |tr ' ' '\n' \
  |grep -m5113060 . \
  |apertium-deshtml \
  |lt-proc nno-dan.automorf.bin \
  |apertium-cleanstream -n \
  |awk 'BEGIN{OFS=FS="\t"} /^\^/{lu++} /\/\*/{u++} END{print "unk","known","tot","cov %";print u,lu-u,lu,100*(lu-u)/lu}'
Direction w/o cmp regular
nob-nno
nno-nob
dan-nob
dan-nno
nno-dan 87.6% 89.0%
nob-dan 89.7% 91.4%
dan-swe 80.6% 83.0%
swe-dan 83.8%
swe-nno
swe-nob
nno-swe
nob-swe