SFST

SFST (Stuttgart Finite State Toolkit) is a set of programs that can be used for writing morphological analysers. It is also one of the possible backends for HFST.

Installation

Prerequisites

On Ubuntu/Debian:

sudo apt-get install libreadline5-dev

Download, compile, install

 wget ftp://ftp.ims.uni-stuttgart.de/pub/corpora/SFST/SFST-1.4.2.tar.gz
 tar -xzvf SFST-1.4.2.tar.gz
 cd SFST/src
 make
 sudo make install
 sudo make libinstall

Note: on Arch Linux, you may have to do make CFLAGS=-fpic instead of make.

Packaged version with fst-proc

A packaged version, with the fst-proc program for processing Apertium input streams can be downloaded from Apertium SVN:

this is now superseded by HFST + hfst-proc, right?

$ svn co http://apertium.svn.sourceforge.net/svnroot/apertium/branches/sfst

Compiling

Follow the standard steps:

$ sh autogen.sh
$ ./configure
$ make
$ make install

Mac users: first do

$ sudo port install libtool 
$ sudo ln -s /opt/local/bin/glibtoolize /bin/libtoolize # or wherever

(also, the call to basename(name) in src/fst-proc.C needs to be changed to eg. name if you want to compile without libiberty sources lying around)

Usage

To try SFST out, you can start by compiling the German transducer, SMOR, that comes with the package:

$ cd data/SMOR
$ make

Wait some time, and you will have a file called smor.a, now you need to compact this so it can be read by fst-proc,

$ fst-compact smor.a smor.ac

Now you can use it,

$ cd ../../src
$ echo "Ich habe ein Bier" |  fst-proc ../data/SMOR/smor.ac
^Ich/<CAP>ich<+PPRO><pers><1><Sg><NoGend><Nom>$ 
^habe/haben<+V><1><Sg><Pres><Konj>/haben<+V><3><Sg><Pres><Konj>/haben<+V><Imp><Sg>/haben<+V><1><Sg><Pres><Ind>$ 
^ein/ein<+ART><Indef><Masc><Nom><Sg>/ein<+ART><Indef><Neut><Nom><Sg>/ein<+ART><Indef><Neut><Akk><Sg>$ ^Bier/*Bier$

It should also work with deformatters and reformatters,

$ echo "Ich habe <em>ein</em> bier" | apertium-deshtml  |  ./fst-proc ../data/SMOR/smor.ac
^Ich/<CAP>ich<+PPRO><pers><1><Sg><NoGend><Nom>$ 
^habe/haben<+V><1><Sg><Pres><Konj>/haben<+V><3><Sg><Pres><Konj>/haben<+V><Imp><Sg>/haben<+V><1><Sg><Pres><Ind>$[ 
<em>]^ein/ein<+ART><Indef><Masc><Nom><Sg>/ein<+ART><Indef><Neut><Nom><Sg>/ein<+ART><Indef><Neut><Akk><Sg>$[<\/em> ]^Bier/*Bier$.[][

Note: fst-proc currently does a rather crude tokenisation based on spaces, so multiwords currently aren't possible.

Morphologies

SFST has the following morphologies available for download:

• Morph-IT! (Italian, 34,968 lemmas, LGPL)
• Omorfi–SFST implementation of word form morphology of Finnish (Finnish, 93,510 lemmas, LGPL)
  • For further information see: Omorfi
• SMOR — comes in the SFST distribution (German, 1,096 lemmas, GPL)
• trmorph (Turkish, wide coverage)
  • For further information see: Turkish

Performance

The analysers produced are fast. For a 1.3Mb analyser (SMOR), it processes ~1,100 words per second. Compare with lttoolbox which processes ~5,000 words per second.

External links

• SFST - Stuttgart Finite State Transducer Tools