Unhammer: Created page with "Say you have a corpus and an analyser, how do you make an trigram frequency list in three shell commands? Grab apertium-cleanstream, then do:

 bzcat corpus.bz2 | ape..."

2015-02-10T13:53:27Z

Created page with "Say you have a corpus and an analyser, how do you make an trigram frequency list in three shell commands? Grab apertium-cleanstream, then do: <pre> bzcat corpus.bz2 | ape..."

New page

Say you have a corpus and an analyser, how do you make an trigram frequency list in three shell commands?

Grab [[apertium-cleanstream]], then do:
<pre>
bzcat corpus.bz2 | apertium-deshtml | lt-proc foo.bin | apertium-cleanstream -n >corpus.ana
paste corpus.ana <(tail -n+1 corpus.ana) <(tail -n+2 corpus.ana) >corpus.trigrams
sort corpus.trigrams | uniq -c | sort -nr > corpus.trigrams.hitparade
</pre>

N-grams - Revision history

Unhammer: Created page with "Say you have a corpus and an analyser, how do you make an trigram frequency list in three shell commands? Grab apertium-cleanstream, then do:
bzcat corpus.bz2 | ape..."