Make a frequency list - Revision history

Unhammer at 10:10, 9 August 2018

2018-08-09T10:10:55Z

Unhammer at 10:07, 9 August 2018

2018-08-09T10:07:35Z

Unhammer: Created page with "The simple way to make a frequency list / hitparade of just forms:

 cat buncha-words.txt | tr ' ' '\n' | sort | uniq -c | sort -nr > hitparade.txt

If you want to ..."

2018-08-09T10:06:28Z

Created page with "The simple way to make a frequency list / hitparade of just forms: <pre> cat buncha-words.txt | tr ' ' '\n' | sort | uniq -c | sort -nr > hitparade.txt </pre> If you want to ..."

New page

The simple way to make a frequency list / hitparade of just forms:
<pre>
cat buncha-words.txt | tr ' ' '\n' | sort | uniq -c | sort -nr > hitparade.txt
</pre>

If you want to make a list of ''lemmas'', you'll have to run it through an analyser. If you have a Spanish corpus in corpus.spa.bz2, then you can get a list of lemmas by analysing it with [[apertium-spa]] and getting the lemmas out with [[Apertium_stream_format#Python_parsing_library|apertium streamparser]] and then doing the sort|uniq|sort dance:

<pre>
#/bin/sh

wget -c https://raw.githubusercontent.com/apertium/streamparser/fff6780c420fdf7437495456d65c9e781a0e437c/streamparser.py

bzcat corpus.spa.bz2 \
| apertium-deshtml \
| apertium -f none -d /path/to/apertium-spa spa-disam \
| python3 -c '
import streamparser
import sys
for lu in streamparser.parse_file(sys.stdin):
print("/".join(set(sub.baseform for r in lu.readings for sub in r if lu.knownness == streamparser.known)))
' \
| LC_ALL=C sort \
| LC_ALL=C uniq -c \
| LC_ALL=C sort -nr > hitparade.txt
</pre>
(the LC_ALL=C makes sorting the same across different Unix locales, avoiding things like treating 'aa' and 'å' the same in Norwegian).

@@ Line 25: / Line 25: @@
 </pre>
 (the LC_ALL=C makes sorting the same across different Unix locales, avoiding things like treating 'aa' and 'å' the same in Norwegian).
+[[Category:Documentation]]
+[[Category:Documentation in English]]

@@ Line 1: / Line 1: @@
+==Forms==
 The simple way to make a frequency list / hitparade of just forms:
 <pre>
 cat buncha-words.txt | tr ' ' '\n' | sort | uniq -c | sort -nr > hitparade.txt
 </pre>
+==Lemmas==
 If you want to make a list of ''lemmas'', you'll have to run it through an analyser. If you have a Spanish corpus in corpus.spa.bz2, then you can get a list of lemmas by analysing it with [[apertium-spa]] and getting the lemmas out with [[Apertium_stream_format#Python_parsing_library|apertium streamparser]] and then doing the sort|uniq|sort dance:
@@ Line 18: / Line 22: @@
 import sys
 for lu in streamparser.parse_file(sys.stdin):
-    print("/".join(set(sub.baseform for r in lu.readings for sub in r if lu.knownness == streamparser.known)))
+    print("/".join(sorted(set(sub.baseform for r in lu.readings for sub in r if lu.knownness == streamparser.known))))
 '                                                         \
     | LC_ALL=C sort                                       \
@@ Line 25: / Line 29: @@
 </pre>
 (the LC_ALL=C makes sorting the same across different Unix locales, avoiding things like treating 'aa' and 'å' the same in Norwegian).
+Note that using <code>spa-disam</code> in the script above will run the disambiguator, so the list will be of disambiguated lemmas. If you don't trust your disambiguator, you may want to run just <code>spa-morph</code>, giving you "lemma-sets" in your hitparade. You can also tweak the streamparser-script to include the main pos if that's useful, etc.
 [[Category:Documentation]]