How to bootstrap a new pair

From Apertium
Revision as of 16:17, 15 January 2021 by Popcorndude (talk | contribs) (remove unnecessary autogen and use apertium-get where possible)
Jump to navigation Jump to search

How to use apertium-init to bootstrap a new language pair (optionally with new monolingual data packages as well).

Prerequisites

You need to get this installed first:

  • apertium/lttoolbox/hfst, see Installation, in particular the prerequisites parts. (You most likely don't need to go all the way, since you should get this stuff from Tino's repositories. If you're on Windows, get the Apertium VirtualBox.)
  • apertium-init.py – put this script in your working directory where you will be downloading language data. You can get the script from https://apertium.org/apertium-init

With two existing monolingual packages

Do the two languages you're making a pair of already have monolingual modules in the repository?

Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages:


First compile the monolingual packages: 

apertium-get XXX
apertium-get YYY

Then generate the pair: 

python3 apertium-init.py XXX-YYY

Then compile the pair: 

cd apertium-XXX-YYY
./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
make -j

And test: 

echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

Now you can add words to apertium-XXX-YYY.XXX-YYY.dix, then test again: 

make -j
echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.

With one existing monolingual package

Does just one of the two languagues you're making a pair of already have a monolingual module in the repository?

Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages; here we assume XXX needs to be made from scratch.
ISO 639-3 codes can be found here: http://www-01.sil.org/iso639-3/codes.asp

First make a new monolingual package: 

python3 apertium-init.py XXX
cd apertium-XXX
make -j
cd ..

Then get and compile the existing monolingual package: 

apertium-get YYY

Then generate the pair: 

python3 apertium-init.py XXX-YYY

Then compile the pair: 

cd apertium-XXX-YYY
./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
make -j

And test: 

echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

Now you can add words to apertium-XXX-YYY.XXX-YYY.dix, then test again: 

make -j
echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.

With no existing monolingual packages

Do none of the two languagues you're making a pair of already have monolingual modules in the repository?

Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages:

First make and compile the new monolingual packages: 

python3 apertium-init.py XXX
cd apertium-XXX
make -j
cd ..

python3 apertium-init.py YYY
cd apertium-YYY
make -j
cd ..

Then generate the pair: 

python3 apertium-init.py XXX-YYY

Then compile the pair: 

cd apertium-XXX-YYY
./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
make -j

And test: 

echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

Now you can add words to apertium-XXX-YYY.XXX-YYY.dix, then test again: 

make -j
echo house | apertium -d . XXX-YYY
echo Haus | apertium -d . YYY-XXX

If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.

HFST and other alternative setups

If you're making a monolingual module that should use HFST/lexc, pass the option --analyser=hfst to apertium-init.py.

If you're making a pair where the "left" side (XXX in the above examples) uses HFST/lexc, pass the option --analyser1=hfst to apertium-init.py.

If you're making a pair where the "right" side (YYY in the above examples) uses HFST/lexc, pass the option --analyser2=hfst to apertium-init.py.

If you're making a pair where the both sides use HFST/lexc, pass the option --analysers=hfst to apertium-init.py.

See https://github.com/apertium/apertium-init for more documentation, or run ./apertium-init.py --help for all options (you can e.g. also make pairs that don't use a statistical disambiguator, or don't use a Constraint Grammar disambiguator).

Choosing an Analyser

Monolingual dictionaries can be set up to use 1 of 3 available formats: monodix, lexc, and lexd.

Monodix is best for languages that have very little conjugation (such as English) or where the conjugation is entirely with suffixes but the stem doesn't change (such as Spanish). Monodix is the default setting for apertium-init or it can be explicitly specified with --analyser=lttoolbox.

Lexc is best for languages where most of the conjugation is done with suffixes, potentially with some stem changes. Apertium-init will generate a Lexc dictionary with the option --analyser=hfst.

Lexd should be used for any language with prefixes, infixes, circumfixes, or other morphology that isn't suffixes. Apertium-init will generate a Lexd dictionary with the option --analyser=lexd.

If you're unsure which one to pick, Lexd will probably work well and generally involves less typing than either Monodix or Lexc.

Retrieved from "https://wiki.apertium.org/w/index.php?title=How_to_bootstrap_a_new_pair&oldid=73037"