Calculating coverage - Revision history

Rocky 734: /* See also= */ fixed a wiki markup

2022-01-10T15:18:35Z

Marcriera: /* Faster coverage testing with frequency lists */

2019-06-10T18:19:23Z

Faster coverage testing with frequency lists

Dharjunior at 10:08, 1 December 2018

2018-12-01T10:08:20Z

Dharjunior at 10:07, 1 December 2018

2018-12-01T10:07:28Z

Dharjunior at 10:05, 1 December 2018

2018-12-01T10:05:23Z

Dharjunior at 00:20, 30 November 2018

2018-11-30T00:20:44Z

Dharjunior at 00:20, 30 November 2018

2018-11-30T00:20:30Z

Sushain at 19:52, 26 April 2018

2018-04-26T19:52:16Z

Francis Tyers: /* More involved scripts */ i don't like supporting this, should be replaced with a simpler example

2017-07-12T10:41:49Z

More involved scripts: i don't like supporting this, should be replaced with a simpler example

Unhammer: /* See also */

2016-01-30T13:27:25Z

@@ Line 117: / Line 117: @@
 '''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see [[Migrating tools to GitHub]].'''
-=See also==
+== See also ==
 * [[Wikipedia dumps]]

@@ Line 108: / Line 108: @@
  $ chmod +x make-freqlist.sh freqlist-coverage.sh
  $ bzcat ~/corpora/nno.txt.bz2 |./make-freqlist.sh > nno.freqlist
- $ <nno.freqlist ./freqlist-coverage.sh -d ~/apertium-svn/languages/apertium-nno/ nno-morph
+ $ < nno.freqlist ./freqlist-coverage.sh -d ~/apertium-svn/languages/apertium-nno/ nno-morph
 ==coverage.py==

@@ Line 11: / Line 11: @@
  sudo cp apertium-cleanstream /usr/local/bin
-'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see Migrating tools to GitHub.'''
+'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see [[Migrating tools to GitHub]].'''
 Then save this as coverage.sh:
@@ Line 115: / Line 115: @@
-'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see Migrating tools to GitHub.'''
+'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see [[Migrating tools to GitHub]].'''
 =See also==

@@ Line 112: / Line 112: @@
 ==coverage.py==
-https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/coverage.py is a coverage script that wraps curl and bzcat (?)
+https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/coverage.py is a coverage script that wraps curl and bzcat.
-==See also==
+'''Note - After Apertium's migration to GitHub, this tool is read-only on the SourceForge repository and does not exist on GitHub. If you are interested in migrating this tool to GitHub, see Migrating tools to GitHub.'''
+=See also==
 * [[Wikipedia dumps]]

← Older revision		Revision as of 10:05, 1 December 2018
Line 1:		Line 1:
	{{Github-migration-check}}

	[[Calculer la couverture\|En français]]		[[Calculer la couverture\|En français]]

← Older revision		Revision as of 19:52, 26 April 2018
Line 1:		Line 1:
			{{Github-migration-check}}

	[[Calculer la couverture\|En français]]		[[Calculer la couverture\|En français]]

@@ Line 110: / Line 110: @@
 https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/coverage.py is a coverage script that wraps curl and bzcat (?)
-==More involved scripts==
-Often it's nice to clean up wikipedia fluff etc. for coverage testing.
-Notes on calculating coverage from wikipedia dumps (based on [[Asturian#Calculating coverage]]).
-(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)
-wikicat.sh:
-<pre>
-#!/bin/sh
-# Clean up wikitext for running through apertium-destxt
-# awk prints full lines, make sure each html element has one
-bzcat "$@" | sed 's/>/>\
-/g' | sed 's/</\
-</g' |\
-# want only stuff between <text...> and </text>
-awk '
-/<text.*>/,/<\/text>/ { print $0 }
-' |\
-sed 's/\./ /g' |\
-# Drop all transwiki links
-sed 's/\[\[\([a-z]\{2,3\}\|bat-smg\|be-x-old\|cbk-zam\|fiu-vro\|map-bms\|nds-nl\|roa-rup\|roa-tara\|simple\|zh-classical\|zh-min-nan\|zh-yue\):[^]]\+\]\]//g' |\
-# wiki markup, retain bar and fie from [[foo|bar]] [[fie]]
-sed 's/\[\[[^]|]*|//g' | sed 's/\]\]//g' | sed 's/\[\[//g' |\
-# wiki markup, retain `bar fie' from [http://foo bar fie] and remove [http://foo]
-sed 's/\[http[^ ]*\([^]]*\)\]/\1/g' |\
-# remove entities
-sed 's/&[^;]*;/ /g' |\
-# and put space instead of punctuation
-sed 's/[;:?,]/ /g' |\
-# Keep only lines starting with a capital letter, removing tables with style info etc.
-grep '^[ 	]*[A-ZÆØÅ]' # Your alphabet here
-</pre>
-count-tokenized.sh:
-<pre>
-#!/bin/sh
-# http://wiki.apertium.org/wiki/Asturian#Calculating_coverage
-# Calculate the number of tokenised words in the corpus:
-apertium-destxt | lt-proc $1 |apertium-retxt |\
-# for some reason putting the newline in directly doesn't work, so two seds
-sed 's/\$[^^]*\^/$^/g' | sed 's/\$\^/$\
-^/g'
-</pre>
-To find all tokens from a wiki dump:
-<pre>
-$ ./wikicat.sh nnwiki-20090119-pages-articles.xml.bz2 > nnwiki.cleaned.txt
-cat nnwiki.cleaned.txt | ./count-tokenized.sh nn-nb.automorf.bin | wc -l
-</pre>
-To find all tokens with at least one analysis (naïve coverage):
-<pre>
-$ cat nnwiki.cleaned.txt  | ./count-tokenized.sh nn-nb.automorf.bin | grep -v '\/\*' | wc -l
-</pre>
-To find the top unknown tokens:
-<pre>
-$ cat nnwiki.cleaned.txt  | ./count-tokenized.sh nn-nb.automorf.bin | sed 's/[ 	]*//g' |\ # tab or space
-   grep '\/\*' | sort -f | uniq -c | sort -gr | head
-</pre>
-=== Script ready to run ===
-corpus-stat.sh
-<pre>
-#!/bin/sh
-# http://wiki.apertium.org/wiki/Asturian#Calculating_coverage
-# Example use:
-# zcat corpa/en.crp.txt.gz | sh corpus-stat.sh
-#CMD="cat corpa/en.crp.txt"
-CMD="cat"
-F=/tmp/corpus-stat-res.txt
-# Calculate the number of tokenised words in the corpus:
-# for some reason putting the newline in directly doesn't work, so two seds
-$CMD | apertium-destxt | lt-proc en-eo.automorf.bin |apertium-retxt | sed 's/\$[^^]*\^/$^/g' | sed 's/\$\^/$\
-^/g' > $F
-NUMWORDS=`cat $F | wc -l`
-echo "Number of tokenised words in the corpus: $NUMWORDS"
-# Calculate the number of words that are not unknown
-NUMKNOWNWORDS=`cat $F | grep -v '\*' | wc -l`
-echo "Number of known words in the corpus: $NUMKNOWNWORDS"
-# Calculate the coverage
-COVERAGE=`calc "round($NUMKNOWNWORDS/$NUMWORDS*1000)/10"`
-echo "Coverage: $COVERAGE %"
-#If you don't have calc, change the above line to:
-#COVERAGE=$(perl -e 'print int($ARGV[0]/$ARGV[1]*1000)/10;' $NUMKNOWNWORDS $NUMWORDS)
-# Show the top-ten unknown words.
-echo "Top unknown words in the corpus:"
-cat $F | grep '\*' | sort -f | uniq -c | sort -gr | head -10
-</pre>
-Sample output:
-<pre>
-$ zcat corpa/en.crp.txt.gz | sh corpus-stat.sh
-Number of tokenised words in the corpus: 478187
-Number of known words in the corpus: 450255
-Coverage: 	94.2 %
-Top unknown words in the corpus:
-^Apollo/*Apollo$
-^Aramaic/*Aramaic$
-^Alberta/*Alberta$
-^de/*de$
-^Abu/*Abu$
-^Bakr/*Bakr$
-^Agassi/*Agassi$
-^Carnegie/*Carnegie$
-^Agrippina/*Agrippina$
-^Achilles/*Achilles$
-^Adelaide/*Adelaide$
-</pre>
 ==See also==

@@ Line 244: / Line 244: @@
 * [[Wikipedia dumps]]
+* [[Cleanstream]]
-* [https://gist.github.com/2283105 apertium-cleanstream]
 [[Category:Documentation]]