User:Ranjan19 - Revision history

Ranjan19 at 19:21, 16 March 2016

2016-03-16T19:21:34Z

Ranjan19: /* Using the zhfst file as a spellchecker and testing it from the command line */

2016-03-15T17:37:10Z

Using the zhfst file as a spellchecker and testing it from the command line

Ranjan19: /* Install language modules */

2016-03-15T15:47:26Z

Install language modules

Ranjan19: Created page with "This guide shows how to compile a speller for an lttoolbox-based analyser and test it on some text. It is possible to use morphologies created in the Apertium platform directl..."

2016-03-15T14:42:45Z

Created page with "This guide shows how to compile a speller for an lttoolbox-based analyser and test it on some text. It is possible to use morphologies created in the Apertium platform directl..."

New page

This guide shows how to compile a speller for an lttoolbox-based analyser and test it on some text. It is possible to use morphologies created in the Apertium platform directly as spellcheckers using the '''libvoikko''' and '''libreoffice-voikko''' extension. We will do exactly that. Apertium hfst transducers can be compiled into libraries that libvoikko can use to perform spell checking, including providing suggestions.
The hfst based languages like Apertium-kaz have already accomplished this task so we'll follow their lead and take a lot of help from their implementation to compile the speller. We will first converting '''lttoolbox to hfst''' and then generating a '''zhfst'''(to be used as a spell-checker) for an lttoolbox-based language such as Hindi(the sample language in this case).

As a prerequisite we need to install libreoffice-voikko on Ubuntu/Debian.This guide shows how to do things by manually compiling most of it manually.

You should be able to do make do with just packages from apt-get, so try that first:
[[Using apertium spellers with libreoffice-voikko on Debian]]. But if one of the packages isn't working, (parts of) this guide may be helpful.

==Install prerequisites==
First add the repository by following [[Prerequisites for Debian]].

Then install speller-prerequisites:

sudo apt-get install libreoffice python3 git make sed findutils zip unzip pkg-config gettext \
libxml++2.6-dev libarchive-dev zlib1g-dev unzip automake autoconf libtool flex bison g++ libreadline-dev hfst

== Manually compile other prerequisites ==

=== hfst-ospell===
wget http://downloads.sourceforge.net/project/hfst/hfst/source/hfstospell-0.4.0.tar.gz
tar xvf hfstospell-0.4.0.tar.gz
cd hfstospell-0.4.0
./configure --enable-zhfst
make -j4
sudo make install

=== Libvoikko===

sudo apt-get install hfst-ospell-dev

wget http://www.puimula.org/voikko-sources/libvoikko/libvoikko-4.0.tar.gz
tar xvf libvoikko-4.0.tar.gz
cd libvoikko-4.0
./autogen.sh
./configure --with-dictionary-path=$HOME/.voikko --enable-hfst
make -j4
sudo make install

You may also have to put <pre>export LD_LIBRARY_PATH=/usr/local/lib</pre> in your ~/.bashrc

=== voikko-fi===
wget http://www.puimula.org/voikko-sources/voikko-fi/voikko-fi-2.0.tar.gz
tar xvf voikko-fi-2.0.tar.gz
cd voikko-fi-2.0
PATH=/usr/local/voikko/bin:$PATH make vvfst
sudo make vvfst-install DESTDIR=/usr/local/voikkodict

==Install language modules==

This is based on [[Minimal installation from SVN]]:

* To install Kazakh language module, first get it. We'll need Kazakh module in the implementation as mentioned earlier.
svn co https://svn.code.sf.net/p/apertium/svn/languages/apertium-kaz/
cd apertium-kaz
./autogen.sh --enable-ospell
make -j4

* To install Hindi language module, first get it. We'll need Kazakh module in the implementation as mentioned earlier.
svn co https://svn.code.sf.net/p/apertium/svn/languages/apertium-hin/
cd apertium-hin
./autogen.sh --enable-ospell
make -j4

=== Converting lttoolbox binary into hfst ===

lt-print hin.autogen.bin | hfst-txt2fst -e ε > hin.autogen.hfst
cd ..

Here `lt-print foo.autogen.bin` gives a .att file which is taken as an input to hfst-txt2fst.

=== Generating a zhfst package from the hfst file generated ===

Here we have observed how apertium-kaz generates the zhfst file and implemented that here.

* Copy “dev/editdist.py” and “speller” from apertium-kaz to apertium-hin.

cp apertium-kaz/dev/editdist.py apertium-hin/dev/editdist.py
cp -r apertium-kaz/speller apertium-hin/speller

* Run the following commands in the Hindi language module. These commands have been taken from apertium-kaz zhfst conversion when we compile apertium-kaz i.e during the `make` of apertium-kaz.

cd apertium-hin
cat hin.autogen.hfst | hfst-fst2fst -t | hfst-project --project=lower | hfst-minimise |hfst-fst2fst -f olw -o acceptor.default.hfst
grep -v -e "^#" -e "^$" speller/words.default.txt | hfst-strings2fst -j -o words.default.hfst
echo "?*;" | hfst-regexp2fst -S -o anystar.hfst
grep -v -e "^#" -e "^$" speller/strings.default.txt | hfst-strings2fst -j | hfst-concatenate anystar.hfst - \
| hfst-concatenate - anystar.hfst -o strings.default.hfst
python dev/editdist.py -v -s -d 1 -e '@0@' -i speller/editdist.default.txt -a acceptor.default.hfst > editdist.default.hfst.txt
hfst-txt2fst -i editdist.default.hfst.txt -e '@0@' -o editdist.default.hfst
rm -f editdist.default.hfst.txt
hfst-disjunct strings.default.hfst editdist.default.hfst | hfst-minimise | hfst-repeat -f 1 -t 2 -o editstrings.default.hfst
hfst-disjunct words.default.hfst editstrings.default.hfst | hfst-fst2fst -f olw -o errmodel.default.hfst
rm -f hin.zhfst
zip -Z store -j hin.zhfst acceptor.default.hfst errmodel.default.hfst speller/index.xml

Here we expect that the proper zhfst file has been generated in the hindi language module.

=== Using the zhfst file as a spellchecker and testing it from the command line ===

There are two ways to go about this.

* Using hfst-ospell

echo "कौशल" | hfst-ospell -S hin.zhfst
echo "आदमी" | hfst-ospell -S hin.zhfst

In the above two cases, since the spellings are correct the output should be “The word is in the lexicon”. Not working. The error comes “word not in the lexicon, cannot provide corrections”.

echo "अगामी" | hfst-ospell -S hin.zhfst

In this case the spelling of the word is incorrect and requires a trivial correction. The output would be the correct options. Not working. The error comes “word not in the lexicon, cannot provide corrections”.

* Using voikkospell

cd apertium-hin
cp hin.zhfst ~/.voikko/3/hin.zhfst
echo “अगामी" | tr ' ' '\n' | voikkospell -d hin.zhfst -s

In this case the spelling of the word is incorrect and requires a trivial correction. The output would be the correct options.

← Older revision		Revision as of 19:21, 16 March 2016
Line 1:		Line 1:
	This guide shows how to compile a speller for an lttoolbox-based analyser and test it on some text. It is possible to use morphologies created in the Apertium platform directly as spellcheckers using the '''libvoikko''' and '''libreoffice-voikko''' extension. We will do exactly that. Apertium hfst transducers can be compiled into libraries that libvoikko can use to perform spell checking, including providing suggestions.		This guide shows how to compile a speller for an lttoolbox-based analyser and test it on some text. It is possible to use morphologies created in the Apertium platform directly as spellcheckers using the '''libvoikko''' and '''libreoffice-voikko''' extension. We will do exactly that. Apertium hfst transducers can be compiled into libraries that libvoikko can use to perform spell checking, including providing suggestions.

	The hfst based languages like Apertium-kaz have already accomplished this task so we'll follow their lead and take a lot of help from their implementation to compile the speller. We will first converting '''lttoolbox to hfst''' and then generating a '''zhfst'''(to be used as a spell-checker) for an lttoolbox-based language such as Hindi(the sample language in this case).		The hfst based languages like Apertium-kaz have already accomplished this task so we'll follow their lead and take a lot of help from their implementation to compile the speller. We will first converting '''lttoolbox to hfst''' and then generating a '''zhfst'''(to be used as a spell-checker) for an lttoolbox-based language such as Hindi(the sample language in this case).

@@ Line 119: / Line 119: @@
  cd apertium-hin
  cp hin.zhfst ~/.voikko/3/hin.zhfst
- echo “अगामी" | tr ' ' '\n' | voikkospell -d hin.zhfst -s
+ echo "अगामी" | tr ' ' '\n' | voikkospell -d hin.zhfst -s
 In this case the spelling of the word is incorrect and requires a trivial correction. The output would be the correct options.

@@ Line 52: / Line 52: @@
 This is based on [[Minimal installation from SVN]]:
-* To install Kazakh language module, first get it. We'll need Kazakh module in the implementation as mentioned earlier.
+* To install Kazakh language module, first get it. We'll need Kazakh module in the implementation as mentioned earlier. We don't need to build it.
  svn co https://svn.code.sf.net/p/apertium/svn/languages/apertium-kaz/
- cd apertium-kaz
- ./autogen.sh --enable-ospell
- make -j4
 * To install Hindi language module, first get it. We'll need Kazakh module in the implementation as mentioned earlier.