Difference between revisions of "Hfst"
m (Text replacement - "(chat|irc)\.freenode\.net" to "irc.oftc.net") |
|||
| (80 intermediate revisions by 12 users not shown) | |||
| Line 1: | Line 1: | ||
{{TOCD}} |
{{TOCD}} |
||
'''hfst''' is the Helsinki finite-state toolkit. This is formalism-compatible with both lexc and twolc, so, kind of like [[foma]] is to xfst. |
'''hfst''' is the Helsinki finite-state toolkit. This is formalism-compatible with both lexc and twolc, so, kind of like [[foma]] is to xfst. It is currently being used in [[apertium-sme-nob]], [[apertium-fin-sme]], [[apertium-kaz-tat]] and in few other pairs which involve Turkic languages. |
||
The IRC channel is <code>#hfst</code> at <code>irc.oftc.net</code> (you may try [irc://irc.oftc.net/#hfst irc://irc.oftc.net/#hfst] if your browser supports it, or enter #hfst into https://webchat.oftc.net/ if you want a web client). The [https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstHome HFST Wiki] has some very good documentation (see especially the page [https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstReadme HfstReadme] when you run into compilation problems). |
|||
==Prerequisites== |
|||
HFST is actually created as a set of wrappers over several possible ''back-ends'', [[Foma]], [[OpenFST]], [[SFST]], …. The latest versions of HFST include the back-ends you need, so there's no reason to install any of these backends separately. |
|||
* automake, autoconf, libtool |
|||
{{Github-migration-check}} |
|||
==Compiling== |
|||
==Building and installing HFST== |
|||
<span style="color: #f00;">See [[Installation]], for most real operating systems you can now get pre-built packages of HFST (as well as other core tools) through your regular package manager.</span> |
|||
===Subversion checkout=== |
|||
:"MacOS X note: you need XCode installed on your Mac. It came with your computer, and can be downloaded from [http://developer.apple.com/ Apple] (registration required)" |
|||
If you wish to hack on the HFST C++ code itself (or you are on some system that doesn't have packages yet), you can follow this procedure: |
|||
===Install prerequisites=== |
|||
You will need the regular build dependencies: |
|||
* <code>automake, autoconf, libtool, flex, bison, g++, libreadline-dev</code> |
|||
If you've already installed apertium/lttoolbox these should be installed already; if not, they should be easily installable with your package manager, e.g. |
|||
* Ubuntu: <code>sudo apt-get install automake autoconf libtool flex bison g++ libreadline-dev</code> |
|||
* Arch Linux: <code>sudo pacman -S base-devel</code> |
|||
* MacOS X users should install the general [[Prerequisites_for_Mac_OS_X]] first, then <code>sudo port install bison readline</code> |
|||
===Download HFST=== |
|||
Either use the latest release (recommended for users), or go with the bleeding-edge Git version (recommended for developers). |
|||
====From Git repository==== |
|||
<pre> |
<pre> |
||
$ |
$ git clone https://github.com/hfst/hfst.git |
||
$ cd |
$ cd hfst/ |
||
$ ./autogen.sh |
|||
$ autoreconf -i |
|||
$ ./configure |
$ ./configure |
||
$ make |
$ make |
||
$ sudo make install |
|||
</pre> |
</pre> |
||
(The autogen step is only needed when using Git, not with the tarball.) |
|||
===Prepackaged tarball=== |
|||
====Released tarball==== |
|||
Download the latest version from [http://sourceforge.net/projects/hfst/files/], and unzip. Then follow the instructions in the README file, i.e.: |
|||
Download the latest release, named something like hfst-X.Y.Z.tar.gz, from https://github.com/hfst/hfst/releases, then |
|||
<pre> |
<pre> |
||
$ |
$ tar -xzf hfst-X.Y.Z.tgz |
||
$ cd hfst-X.Y.Z/ |
|||
$ ./configure |
|||
$ make |
|||
$ sudo make install |
|||
</pre> |
</pre> |
||
(replacing X.Y.Z for the version you downloaded) |
|||
===Configure=== |
|||
In the configure step, you can turn on/off features and backends and such. <small>The [[OpenFST]] backend is included in the HFST distribution, while [[foma]] and [[SFST]] are not and are not recommended since they typically lead to more trouble than it's worth.</small> |
|||
For most users, this should work: |
|||
<pre> |
|||
$ ./configure --enable-proc --without-foma --enable-lexc --enable-all-tools |
|||
</pre> |
|||
The above command will configure it to be installed to /usr/local in the <code>make install</code> step (below). |
|||
If you want hfst and back-ends installed somewhere else, you can do |
|||
<pre> |
|||
$ ./configure --enable-proc --without-foma --enable-lexc --enable-all-tools --prefix=/home/USERNAME/local/ |
|||
</pre> |
|||
'''Note: When we say USERNAME we mean your username, you need to replace it with your username, if you don't know what it is, you can find out by typing <code>whoami</code>''' |
|||
You can also add <code>--with-unicode-handler=glib</code> (or <code>--with-unicode-handler=ICU</code>) to the ./configure step if you have glib (or ICU) installed and want better Unicode [https://en.wikipedia.org/wiki/Case_folding#Case_folding Case_folding]. |
|||
===Compile and install=== |
|||
If your autotools version is older than 1.14 (check with <code>automake --version</code>), first do: |
|||
<pre>$ scripts/generate-cc-files.sh</pre> |
|||
Build by running |
|||
<pre>$ make</pre> |
|||
Then you need to install (Note: you need to use <code>sudo make install</code> if you installed it in /usr/local (or did not give a --prefix in the configure step); otherwise, no sudo!) |
|||
<pre> |
|||
$ make install |
|||
</pre> |
|||
And finally, unless you have a Mac, you may need to do: |
|||
<pre> |
|||
$ sudo ldconfig |
|||
</pre> |
|||
==Troubleshooting== |
|||
When doing "make" with old autotools (pre 1.14?) |
|||
<pre>make[5]: *** No rule to make target `xre_parse.hh', needed by `xre_lex.ll'. Stop.</pre> |
|||
Run <code>scripts/generate-cc-files.sh</code> and then make again. |
|||
If, during the ./configure step, you see<pre>checking for GNU libc compatible malloc... no |
|||
[…] |
|||
checking for GNU libc compatible realloc... no</pre> and then during make a bunch of errors like: <pre>/usr/local/include/sfst/mem.h:37:57: error: 'malloc' was not declared in this scope</pre>, try the following: |
|||
<pre>sudo ldconfig |
|||
export LD_LIBRARY_PATH=/usr/local/lib |
|||
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig</pre> |
|||
and then ./configure and make. |
|||
If, during make, you see errors like |
|||
<pre>xre_parse.cc:2293:24: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]</pre> |
|||
try instead |
|||
<pre> |
|||
make CXXFLAGS=-fpermissive |
|||
</pre> |
|||
If, when compiling a dictionary, you end up in a "foma" prompt where you can type stuff, you should remove anything related to foma or "hfst-xfst" from your system, and build HFST anew as described above. |
|||
For more advices on installation problems, have a look at [https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstReadme the Hfst Readme page]. |
|||
See also [[Foma]], [[OpenFST]] and [[SFST]] for problems regarding the back-ends. |
|||
==Using== |
==Using== |
||
<pre> |
<pre> |
||
$ svn co https://victorio.uit.no/langtech/trunk/ |
$ svn co https://victorio.uit.no/langtech/trunk/langs/fao |
||
$ cd fao/src |
$ cd fao/src |
||
$ make -f Makefile.hfst |
$ make -f Makefile.hfst |
||
| Line 85: | Line 173: | ||
</pre> |
</pre> |
||
== |
==See also== |
||
<pre> |
|||
<jacobEo> why use that? |
|||
<spectie> because it has a really expressive formalism for languages with complex morphology, like finnish, sami and basque |
|||
<jacobEo> could you give an example of the most important thing it can do that lttoolbox cant? |
|||
<spectie> stem internal variation |
|||
<jacobEo> and how does it do that? |
|||
<spectie> by composing different transducers |
|||
<spectie> jacobEo, e.g. you have your lexical transducer, then you have your phonological transducer and you compose the two |
|||
<spectie> jacobEo, it's like the postgeneration in apertium, but much more integrated |
|||
<jacobEo> so if we imaging good, better ,best was called good, geod, gyod and that was a rule for all adjectives? |
|||
<jacobEo> like that? |
|||
<jacobEo> i mean like a paradigm |
|||
<jacobEo> saying o=<pst> |
|||
<jacobEo> saying e=<cmp> |
|||
<jacobEo> saying y=<sup> |
|||
<jimregan> jacobEo, yes |
|||
<spectie> it's like taking care of "live+ed" --> "lived" |
|||
<spectie> instead of having two paradigms for "live" and "jump" you would have one paradigm |
|||
<spectie> +ed |
|||
<spectie> then you would have a phonological rule that says "at morpheme boundaries, collapse ee -> e |
|||
<jacobEo> k |
|||
* [[Starting a new language with HFST]] |
|||
<jacobEo> so its much slower, I suppose |
|||
<jimregan> nah |
|||
<spectie> jacobEo, the compilation is slower |
|||
<jimregan> slower to compile, sure |
|||
<jimregan> you provide definitions for things like what is a vowel, what is a consonant, and where umlauting happens and what it is |
|||
<jimregan> ...in nightmarish syntax that escaped from the 70s |
|||
<spectie> http://paste2.org/p/532099 |
|||
LowerG2 = [ |
|||
[ |
|||
(Cns:0) LCnsPhon7 (Cns:0) LCnsPhon (Cns:0) ! xy that cannot be G3, since x cannot form xy G3. |
|||
| ! nijbe, |
|||
[ ! This section is for 3-cons G2. and for 2cns G2 that share the initial cns with 2cns G3 |
|||
(Cns:0) [:j|:l|:m|:n|:v] (Cns:0) :s :t ! S9, 3cns-G2 bäjstov, etc. |
|||
| |
|||
(Cns:0) [ :l | :r | :n | :j ] (Cns:0) :s :k ! S9, 3cns-G2 sválskes, etc. |
|||
| |
|||
(Cns:0) [ ! 2cns G2 that share the initial cns wit 2cns G3 |
|||
:b (Cns:0) [ :d | :m | :j | :l | :n | :n :j | :r | :s | :t :j | :t :s ] ! S9, initial b |
|||
| ! gábdev |
|||
:d (Cns:0) [ :j | :n | :n :j ] ! S7, initial d |
|||
| ! iednev |
|||
:g (Cns:0) :ŋ ! S7, initial g |
|||
| ! låg0ŋot |
|||
:k (Cns:0) [ :n | :k ] ! S7, initial k |
|||
| |
|||
:g (Cns:0) :n !däggna:degna |
|||
] ! |
|||
| |
|||
:r (Cns:0) :s :j :t ! S9, rsjt, bårsjtav |
|||
] |
|||
] |
|||
- [ |
|||
[ d t [s|j] ] | b b | d d | g g | k [ s | t | t j | t s ] | |
|||
f ':0 f | l ':0 l | m ':0 m | n ':0 n | n ':0 n j | ŋ ':0 ŋ | r ':0 r | s ':0 s | s ':0 s j | v ':0 v |
|||
] |
|||
]; |
|||
<spectie> the formalism is human-hostile |
|||
<Unhammer> ^^^ and sed-hostile |
|||
</pre> |
|||
==External links== |
==External links== |
||
* [http://www.ling.helsinki.fi/kieliteknologia/tutkimus/hfst/ Helsinki Finite-State Transducer Technology (HFST)] |
* [http://www.ling.helsinki.fi/kieliteknologia/tutkimus/hfst/ Helsinki Finite-State Transducer Technology (HFST)] |
||
[[Category:Morphological analysers]] |
[[Category:Morphological analysers]] |
||
[[Category:HFST]] |
|||
Latest revision as of 06:25, 27 May 2021
hfst is the Helsinki finite-state toolkit. This is formalism-compatible with both lexc and twolc, so, kind of like foma is to xfst. It is currently being used in apertium-sme-nob, apertium-fin-sme, apertium-kaz-tat and in few other pairs which involve Turkic languages.
The IRC channel is #hfst at irc.oftc.net (you may try irc://irc.oftc.net/#hfst if your browser supports it, or enter #hfst into https://webchat.oftc.net/ if you want a web client). The HFST Wiki has some very good documentation (see especially the page HfstReadme when you run into compilation problems).
HFST is actually created as a set of wrappers over several possible back-ends, Foma, OpenFST, SFST, …. The latest versions of HFST include the back-ends you need, so there's no reason to install any of these backends separately.
This page is out of date as a result of the migration to GitHub. Please update this page with new documentation and remove this warning. If you are unsure how to proceed, please contact the GitHub migration team.
Building and installing HFST[edit]
See Installation, for most real operating systems you can now get pre-built packages of HFST (as well as other core tools) through your regular package manager.
If you wish to hack on the HFST C++ code itself (or you are on some system that doesn't have packages yet), you can follow this procedure:
Install prerequisites[edit]
You will need the regular build dependencies:
automake, autoconf, libtool, flex, bison, g++, libreadline-dev
If you've already installed apertium/lttoolbox these should be installed already; if not, they should be easily installable with your package manager, e.g.
- Ubuntu:
sudo apt-get install automake autoconf libtool flex bison g++ libreadline-dev - Arch Linux:
sudo pacman -S base-devel - MacOS X users should install the general Prerequisites_for_Mac_OS_X first, then
sudo port install bison readline
Download HFST[edit]
Either use the latest release (recommended for users), or go with the bleeding-edge Git version (recommended for developers).
From Git repository[edit]
$ git clone https://github.com/hfst/hfst.git $ cd hfst/ $ ./autogen.sh $ ./configure $ make
(The autogen step is only needed when using Git, not with the tarball.)
Released tarball[edit]
Download the latest release, named something like hfst-X.Y.Z.tar.gz, from https://github.com/hfst/hfst/releases, then
$ tar -xzf hfst-X.Y.Z.tgz $ cd hfst-X.Y.Z/
(replacing X.Y.Z for the version you downloaded)
Configure[edit]
In the configure step, you can turn on/off features and backends and such. The OpenFST backend is included in the HFST distribution, while foma and SFST are not and are not recommended since they typically lead to more trouble than it's worth.
For most users, this should work:
$ ./configure --enable-proc --without-foma --enable-lexc --enable-all-tools
The above command will configure it to be installed to /usr/local in the make install step (below).
If you want hfst and back-ends installed somewhere else, you can do
$ ./configure --enable-proc --without-foma --enable-lexc --enable-all-tools --prefix=/home/USERNAME/local/
Note: When we say USERNAME we mean your username, you need to replace it with your username, if you don't know what it is, you can find out by typing whoami
You can also add --with-unicode-handler=glib (or --with-unicode-handler=ICU) to the ./configure step if you have glib (or ICU) installed and want better Unicode Case_folding.
Compile and install[edit]
If your autotools version is older than 1.14 (check with automake --version), first do:
$ scripts/generate-cc-files.sh
Build by running
$ make
Then you need to install (Note: you need to use sudo make install if you installed it in /usr/local (or did not give a --prefix in the configure step); otherwise, no sudo!)
$ make install
And finally, unless you have a Mac, you may need to do:
$ sudo ldconfig
Troubleshooting[edit]
When doing "make" with old autotools (pre 1.14?)
make[5]: *** No rule to make target `xre_parse.hh', needed by `xre_lex.ll'. Stop.
Run scripts/generate-cc-files.sh and then make again.
If, during the ./configure step, you see
checking for GNU libc compatible malloc... no […] checking for GNU libc compatible realloc... no
and then during make a bunch of errors like:
/usr/local/include/sfst/mem.h:37:57: error: 'malloc' was not declared in this scope
, try the following:
sudo ldconfig export LD_LIBRARY_PATH=/usr/local/lib export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
and then ./configure and make.
If, during make, you see errors like
xre_parse.cc:2293:24: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]
try instead
make CXXFLAGS=-fpermissive
If, when compiling a dictionary, you end up in a "foma" prompt where you can type stuff, you should remove anything related to foma or "hfst-xfst" from your system, and build HFST anew as described above.
For more advices on installation problems, have a look at the Hfst Readme page.
See also Foma, OpenFST and SFST for problems regarding the back-ends.
Using[edit]
$ svn co https://victorio.uit.no/langtech/trunk/langs/fao $ cd fao/src $ make -f Makefile.hfst $ echo "orð" | hfst-lookup ../bin/fao-morph.hfst lookup> orð orð+N+Neu+Sg+Nom+Indef orð orð+N+Neu+Sg+Acc+Indef orð orð+N+Neu+Pl+Nom+Indef orð orð+N+Neu+Pl+Acc+Indef lookup> $
To compile lexc code, first concatenate all the lexc files:
$ cat fao-lex.txt noun-fao-lex.txt noun-fao-morph.txt adj-fao-lex.txt \ adj-fao-morph.txt verb-fao-lex.txt verb-fao-morph.txt adv-fao-lex.txt \ abbr-fao-lex.txt acro-fao-lex.txt pron-fao-lex.txt punct-fao-lex.txt \ numeral-fao-lex.txt pp-fao-lex.txt cc-fao-lex.txt cs-fao-lex.txt \ interj-fao-lex.txt det-fao-lex.txt > ../tmp/lexc-all.txt
To compile this, just use the hfst-lexc program,
hfst-lexc < ../tmp/lexc-all.txt > ../bin/lexc-fao.bin
To compile the twol rules, just use the hfst-twolc program,
$ hfst-twolc twol-fao.txt > twol-fao.bin
And then to compose the lexicon and rule file, use hfst-compose-intersect:
$ hfst-compose-intersect -l lexc-fao.bin twol-fao.bin -o fao-gen.hfst
This will create a generator, if you want an analyser, you just need to invert the generator with hfst-invert:
$ hfst-invert fao-gen.hfst -o fao-morph.hfst
