Setup for working on morphological dictionaries - Revision history

Francis Tyers: /* Example */

2013-11-19T17:59:47Z

Example

Francis Tyers at 15:09, 19 November 2013

2013-11-19T15:09:47Z

Francis Tyers: /* Example */

2013-11-19T12:19:58Z

Example

Francis Tyers: /* Example */

2013-11-19T12:18:55Z

Example

Francis Tyers: /* Example */

2013-11-19T12:18:20Z

Example

Francis Tyers at 12:06, 19 November 2013

2013-11-19T12:06:47Z

Francis Tyers: Created page with "The most important thing when working on morphological dictionaries is to add words by frequency, and be able to check your morphology against a corpus. You will need: * a W..."

2013-11-19T12:05:56Z

Created page with "The most important thing when working on morphological dictionaries is to add words by frequency, and be able to check your morphology against a corpus. You will need: * a W..."

New page

The most important thing when working on morphological dictionaries is to add words by frequency, and be able to check your morphology against a corpus.

You will need:

* a Wikipedia dump (see [http://dumps.wikimedia.org here]), (the pages-articles.xml.bz2 file) and [[Wikipedia Extractor]].
* an Apertium monolingual language directory.

==Example==

Let's suppose we want to make an Urdu corpus.

===Step 1===

Download the Apertium Urdu module:

<pre>
$ svn co https://svn.code.sf.net/p/apertium/svn/languages/apertium-urd
</pre>

Compile it in the usual way.

Next, make a directory for your corpus.

<pre>
$ mkdir -p urdu/wikipedia

$ cd urdu/wikipedia
</pre>

Now, download the corpus, and extract the text:

<pre>
$ wget http://dumps.wikimedia.org/urwiki/20131109/urwiki-20131109-pages-articles.xml.bz2

$ mkdir output

$ bzcat2 urwiki-20131109-pages-articles.xml.bz2| python WikiExtractor.py -o output/

$ cat output/*/* | strip_html.py > urd.crp.txt
</pre>

Note: <code>strip_html.py</code> is a script that removes text between <code><</code> and <code>></code> from a text file.

After you have your corpus you can generate your frequency list:

<pre>
$ cat urd.crp.txt | apertium-destxt | lt-proc /path/to/urd.automorf.bin | apertium-retxt | sed 's/\$\W*\^/$\n^/g' |\
cut -f2 -d'^' | cut -f1 -d'/' | sort -f | uniq -c | sort -gr | grep -v '[0-9] [0-9]' > urd.hitparade.txt
</pre>

You should also make a small script file called <code>hitparade.sh</code> in this directory. It should contain:

<pre>
cat urd.hitparade.txt | apertium-destxt | lt-proc /path/to/urd.automorf.bin | apertium-retxt
</pre>

[[Category:Development]]

@@ Line 42: / Line 42: @@
 $ cat output/*/* | strip_html.py > urd.crp.txt
+$ rm -r output/
 </pre>

@@ Line 65: / Line 65: @@
 DIX=/path/to/apertium-urd/apertium-urd.urd.dix
 BIN=/path/to/apertium-urd/urd.automorf.bin
+LANG=urd
-cat bel.crp.txt | cut -f2 | grep -v '>(' | sed 's/&lt;/</g' | sed 's/&gt;/>/g' | apertium-destxt | lt-proc $BIN |\
+cat $LANG.crp.txt | cut -f2 | grep -v '>(' | sed 's/&lt;/</g' | sed 's/&gt;/>/g' | apertium-destxt | lt-proc $BIN |\
- apertium-retxt | sed 's/\$\W*\^/$\n^/g' > /tmp/urd.coverage.txt
+ apertium-retxt | sed 's/\$\W*\^/$\n^/g' > /tmp/$LANG.coverage.txt
 EDICT=`cat $DIX | grep -e '<e lm' | wc -l`;
 EPAR=`cat $DIX | grep '<pardef ' | wc -l`;
-TOTAL=`cat /tmp/urd.coverage.txt | wc -l`
+TOTAL=`cat /tmp/$LANG.coverage.txt | wc -l`
-KNOWN=`cat /tmp/urd.coverage.txt | grep -v '*' | wc -l`
+KNOWN=`cat /tmp/$LANG.coverage.txt | grep -v '*' | wc -l`
 COV=`calc $KNOWN / $TOTAL`;
 DATE=`date`;

@@ Line 16: / Line 16: @@
 </pre>
-Compile it in the usual way.
+Compile it in the usual way:
+<pre>
+$ ./autogen.sh
+$ ./configure
+$ make
+</pre>
 Next, make a directory for your corpus.

@@ Line 51: / Line 51: @@
 <pre>
 cat urd.hitparade.txt | apertium-destxt | lt-proc /path/to/urd.automorf.bin | apertium-retxt
+</pre>
+You will also want a script called <code>coverage.sh</code>, it should look something like:
+<pre>
+DIX=/path/to/apertium-urd/apertium-urd.urd.dix
+BIN=/path/to/apertium-urd/urd.automorf.bin
+cat bel.crp.txt | cut -f2 | grep -v '>(' | sed 's/&lt;/</g' | sed 's/&gt;/>/g' | apertium-destxt | lt-proc $BIN |\
+ apertium-retxt | sed 's/\$\W*\^/$\n^/g' > /tmp/urd.coverage.txt
+EDICT=`cat $DIX | grep -e '<e lm' | wc -l`;
+EPAR=`cat $DIX | grep '<pardef ' | wc -l`;
+TOTAL=`cat /tmp/urd.coverage.txt | wc -l`
+KNOWN=`cat /tmp/urd.coverage.txt | grep -v '*' | wc -l`
+COV=`calc $KNOWN / $TOTAL`;
+DATE=`date`;
+echo -e $DATE"\t"$EPAR":"$EDICT"\t"$KNOWN"/"$TOTAL"\t"$COV >> history.log
+tail -1 history.log
+</pre>
+And a further file called <code>new-parade.sh</code>:
+<pre>
+cat /tmp/urd.coverage.txt | cut -f2 -d'^' | cut -f1 -d'/' | sort -f | uniq -c | sort -gr  | grep -v '[0-9] [0-9]' > urd.hitparade.txt
 </pre>

@@ Line 39: / Line 39: @@
 $ mkdir output
-$ bzcat2 urwiki-20131109-pages-articles.xml.bz2| python WikiExtractor.py -o output/
+$ bzcat urwiki-20131109-pages-articles.xml.bz2| python WikiExtractor.py -o output/
 $ cat output/*/* | strip_html.py > urd.crp.txt

@@ Line 9: / Line 9: @@
 Let's suppose we want to make an Urdu corpus.
-===Step 1===
 Download the Apertium Urdu module: