Difference between revisions of "Calculating coverage"
(→See also=: fixed a wiki markup) |
|||
Line 117: | Line 117: | ||
'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see [[Migrating tools to GitHub]].''' |
'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see [[Migrating tools to GitHub]].''' |
||
=See also== |
== See also == |
||
* [[Wikipedia dumps]] |
* [[Wikipedia dumps]] |
Latest revision as of 15:18, 10 January 2022
Contents
Simple bidix-trimmed coverage testing[edit]
First install apertium-cleanstream:
svn checkout https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/apertium-cleanstream cd apertium-cleanstream make sudo cp apertium-cleanstream /usr/local/bin
Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see Migrating tools to GitHub.
Then save this as coverage.sh:
#!/bin/bash mode=$1 outfile=/tmp/$mode.clean apertium -d . $mode | apertium-cleanstream -n > $outfile total=$(grep -c '^\^' $outfile) unknown=$(grep -c '/\*' $outfile) bidix_unknown=$(grep -c '/@' $outfile) known_percent=$(calc -p "round( 100*($total-$unknown-$bidix_unknown)/$total, 3)") echo "$known_percent % known tokens ($unknown unknown, $bidix_unknown bidix-unknown of total $total tokens)" echo "Top unknown words:" grep '/[*@]' $outfile | sort | uniq -c | sort -nr | head
And run it like
cat asm.corpus | bash coverage.sh asm-eng-biltrans
(The bidix-unknown count should always be 0 if your pair uses automatic analyser trimming.)
TODO: paradigm-coverage (less naïve)[edit]
On an analysed corpus, we can sum frequencies into bins for each lemma+mainpos, so if the analysed corpus contains
musa/mus<n><f><sg><def>/muse<vblex><past> mus/mus<n><f><sg><ind>/mus<n><f><pl><ind>/muse<vblex><imp> musene/mus<n><f><pl><def>
then output has
3 mus<n><f> 2 muse<vblex>
and we can find paradigms that are likely to mess up disambiguation, or where we need to ensure that the bidix contains the highest-frequency paradigm (since the bidix is typically smaller than the monodix).
We could also weight these numbers by number of unique forms in the pardef; if the verb pardef has 6 unique forms and then noun only 3, then the above output should be even more skewed:
0.33 mus<n><f> 0.75 muse<vblex>
Faster coverage testing with frequency lists[edit]
If words appear several times in your corpus, why bother analysing them several times? We can make a frequency list first and add together the frequencies. This script does some very stupid tokenisation and creates a frequency list:
make-freqlist.sh:
#!/bin/bash if [[ -t 0 ]]; then echo "Expecting a corpus on stdin" exit 2 fi tr '[:space:][:punct:]' '\n' | grep . | sort | uniq -c | sort -nr
And this script runs your analyser, summing up the frequencies:
freqlist-coverage.sh:
#!/bin/bash set -e -u if [[ $# -eq 0 || -t 0 ]]; then echo "Expecting apertium arguments and a 'sort|uniq -c|sort -nr' style frequency list on stdin" echo "For example:" echo "\$ < spa.freqlist $0 -d . spa-morph" exit 2 fi sed 's%^ *%<apertium-notrans>%;s% %</apertium-notrans>%;s%$% .%' | apertium -f html-noent "$@" | awk -F'</?apertium-notrans>| *\\^\\./\\.<sent><clb>\\$' ' /[/][*@]/ { unknown+=$2 if(!printed) print "Top unknown tokens:" if(++printed<10) print $2,$3 next } { known+=$2 } END { total=known+unknown known_pct=100*known/total unk_pct=100*unknown/total print known_pct" % known of total "total" tokens" }'
Usage:
$ chmod +x make-freqlist.sh freqlist-coverage.sh $ bzcat ~/corpora/nno.txt.bz2 |./make-freqlist.sh > nno.freqlist $ < nno.freqlist ./freqlist-coverage.sh -d ~/apertium-svn/languages/apertium-nno/ nno-morph
coverage.py[edit]
https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/coverage.py is a coverage script that wraps curl and bzcat.
Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see Migrating tools to GitHub.