Analysing Finnish text

Installation

First make a directory called something like "source" in your home directory. The commands below assume you start in that directory.

Install SFST

$ sudo apt-get install libreadline5-dev
$ wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/SFST/SFST-1.4.2.tar.gz
$ cd SFST/src
$ make
$ sudo make install
$ sudo make libinstall
$ cd ..

Install OpenFST

$ wget http://openfst.cs.nyu.edu/twiki/pub/FST/FstDownload/openfst-1.2.6.tar.gz
$ tar -xzvf openfst-1.2.6.tar.gz 
$ cd openfst-1.2.6/
$ ./configure
$ make
$ sudo make install
$ cd ..

Install HFST

$ wget http://downloads.sourceforge.net/project/hfst/hfst/hfst-2.4.1.tar.gz
$ tar -xzvf hfst-2.4.1.tar.gz 
$ cd hfst-2.4.1/
$ ./configure
$ make 
$ sudo make install
$ cd ..

Install Omorfi

$ svn co http://svn.gna.org/svn/omorfi/trunk omorfi
$ cd omorfi
$ sh autogen.sh
$ ./configure 
$ make
$ sudo make install
$ cd ..

Install VISLCG

Some hints can be found here:

• Giellatekno's vislcg3 installation page
• Giellatekno's vislcg3 usage page

The main vislcg3 page is at http://beta.visl.sdu.dk/cg3.html.

Usage

Morphological analysis

Testing

$ cd omorfi/src
$ echo "auton" | hfst-optimized-lookup mor-omorfi.cg.hfst.ol
auton	auto+N+Sg+Gen
$ echo "autojen" | hfst-optimized-lookup mor-omorfi.cg.hfst.ol
autojen	auto+N+Pl+Gen

Morphological disambiguation

The analysis chain is as follows:

1. take text
2. preprocess: change it to one token per line (with proper punctuation handling)
3. morphological analysis
4. change it from omorfi output to vislcg3 input
5. run it through vislcg3

$ echo "Alussa loi Jumala taivaan ja maan." |  hfst-proc -C fin-sme.automorf.hfst   | vislcg3 --trace -g apertium-sme-fin.fin-sme.rlx 
VISL CG-3 Disambiguator version 0.9.7.6378
Codepage: default UTF-8, input UTF-8, output UTF-8, grammar UTF-8
Parsing grammar took 0.06 seconds.
Grammar has 6 sections, 0 templates, 1379 rules, 1312 sets, 748 c-tags, 1299 s-tags.
"<Alussa>"
	"alku" N Sg Ine 
"<loi>"
	"luoda" V Act Ind Prt Sg3 @+FMAINV MAP:2859 
"<Jumala>"
	"Jumala" N Prop Sg Nom @→N MAP:2694 
	"jumala" N Sg Nom @→N MAP:2694 
"<taivaan>"
	"taivas" N Sg Gen @→N MAP:2680 
"<ja>"
	"ja" CC @CNP MAP:2651 
;	"ja" Pcle REMOVE:820 
"<maan>"
	"maa" N Sg Gen @←OBJ MAP:2785 
"<.>"
	"." Punct CLB ADD:793