Difference between revisions of "Calculating coverage"

From Apertium
Jump to navigation Jump to search
(New page: Notes on calculating coverage from wikipedia dumps (based on Asturian#Calculating coverage). wikicat.sh: <pre> #!/bin/sh # clean up wiki for running through apertium-destxt # awk pri...)
 
m
Line 1: Line 1:
Notes on calculating coverage from wikipedia dumps (based on [[Asturian#Calculating coverage]]).
+
Notes on calculating coverage from wikipedia dumps (based on [[Asturian#Calculating coverage]]).
  +
  +
(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)
   
 
wikicat.sh:
 
wikicat.sh:

Revision as of 18:27, 25 March 2009

Notes on calculating coverage from wikipedia dumps (based on Asturian#Calculating coverage).

(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)

wikicat.sh:

#!/bin/sh
# clean up wiki for running through apertium-destxt

# awk prints full lines, make sure each html elt has one
bzcat "$@" | sed 's/>/>\
/g' | sed 's/</\
</g' |\
# want only stuff between <text...> and </text>
awk '
/<text.*>/,/<\/text>/ { print $0 }
' |\
sed 's/\./ /g' |\
sed 's/\[\[[^|]*|//g' | sed 's/\]\]//g' | sed 's/\[\[//g' |\
# wiki markup, retain bar and fie from [[foo|bar]] [[fie]]
sed 's/&.*;/ /g' |\
# remove entities
sed 's/[;:?,]/ /g' |\
# and put space instead of punctuation
grep '^[ 	]*[A-ZÆØÅ]' # Your alphabet here
# Keep only lines starting with a capital letter, removing tables with style info etc.

count-tokenized.sh:

#!/bin/sh
# http://wiki.apertium.org/wiki/Asturian#Calculating_coverage

# Calculate the number of tokenised words in the corpus:
apertium-destxt | lt-proc $1 |apertium-retxt |\
# for some reason putting the newline in directly doesn't work, so two seds
sed 's/\$[^^]*\^/$^/g' | sed 's/\$\^/$\
^/g' 

To find all tokens from a wiki dump:

$ ./wikicat.sh nnwiki-20090119-pages-articles.xml.bz2 | ./count-correct.sh nn-nb.automorf.bin | wc -l

To find all correct tokens:

$ ./wikicat.sh nnwiki-20090119-pages-articles.xml.bz2 | ./count-correct.sh nn-nb.automorf.bin | grep -v '\/\*' | wc -l