How to bootstrap a new pair - Revision history

Kamush: extra info about make langs

2021-04-20T15:30:33Z

extra info about make langs

Popcorndude: add note about dealing with taggers

2021-04-19T22:23:55Z

add note about dealing with taggers

Popcorndude: less redundant instructions, also use apertium-get

2021-01-15T16:32:34Z

less redundant instructions, also use apertium-get

Popcorndude: remove unnecessary autogen and use apertium-get where possible

2021-01-15T16:17:38Z

remove unnecessary autogen and use apertium-get where possible

Popcorndude: /* HFST and other alternative setups */ give some guidance on selecting an analyser (only slightly biased towards lexd)

2021-01-15T05:19:38Z

HFST and other alternative setups: give some guidance on selecting an analyser (only slightly biased towards lexd)

Popcorndude at 21:00, 13 January 2021

2021-01-13T21:00:09Z

Popcorndude: update download link for -init

2020-09-17T16:33:01Z

update download link for -init

Hectoralos: /* Prerequisites */

2020-02-24T05:35:19Z

Prerequisites

Firespeaker: /* Prerequisites */

2018-10-27T21:50:37Z

Prerequisites

Shardulc: GitHub migration

2018-03-09T03:18:37Z

GitHub migration

@@ Line 62: / Line 62: @@
 If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.
+Another useful piece of information about "make langs": itʻs also possible to speed the process by "make -j8 langs", where 8 is replaced by number of CPU cores your computer has.
 ==Checking the tagger==

@@ Line 62: / Line 62: @@
 If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.
+==Checking the tagger==
+Unfortunately, <code>apertium-tagger</code> will crash if given the wrong arguments. This will generally result in no output being produced when you try to translate something. Currently the best way to deal with this is to open the <code>modes.xml</code> files in the two monolingual and checking what options they give in their <code>XXX-tagger</code> and <code>YYY-tagger</code> modes.
+If you find <code>-x</code>, <code>-u 1</code>, <code>-u 2</code>, or <code>-u 3</code> after <code>apertium-tagger</code>, add it to the corresponding line of the <code>modes.xml</code> file in the bilingual directory.
+If you just initialized the monolingual directory, add <code>-u 2</code> in both places.
+If the monolingual <code>modes.xml</code> does not mention <code>apertium-tagger</code>, bootstrap the pair again but add the argument <code>--no-prob1</code> or <code>--no-prob2</code>. If you have made changes that you don't want overwritten, add the argument <code>--rebuild</code> as well, otherwise it will be simplest to delete the entire directory.
 ==HFST and other alternative setups==

@@ Line 9: / Line 9: @@
 * [[apertium-init]].py – put this script in your working directory where you will be downloading language data. You can get the script from https://apertium.org/apertium-init
-==With two existing monolingual packages==
+==Getting the monolingual packages==
-Do the two languages you're making a pair of already have monolingual modules in [https://apertium.github.io/apertium-on-github/source-browser.html the repository]?
+For each of the two languages of the pair, if it already exists, you can download and compile it by running
-Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages:
-First compile the monolingual packages:
 <pre>
 apertium-get XXX
-apertium-get YYY
 </pre>
+Where XXX is the [http://www-01.sil.org/iso639-3/codes.asp ISO 639-3] code of the language.
-Then generate the pair:
-<pre>
-python3 apertium-init.py XXX-YYY
-</pre>
+If the module doesn't exist (if it doesn't appear on [https://apertium.github.io/apertium-on-github/source-browser.html this list] or apertium-get can't find it) then you can create it like this:
-Then compile the pair:
 <pre>
+# bootstrap the module
-cd apertium-XXX-YYY
-./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
-make -j
-</pre>
-And test:
-<pre>
-echo house | apertium -d . XXX-YYY
-echo Haus | apertium -d . YYY-XXX
-</pre>
-Now you can add words to apertium-XXX-YYY.XXX-YYY.dix, then test again:
-<pre>
-make -j
-echo house | apertium -d . XXX-YYY
-echo Haus | apertium -d . YYY-XXX
-</pre>
-If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.
-==With one existing monolingual package==
-Does just one of the two languagues you're making a pair of already have a monolingual module in [https://apertium.github.io/apertium-on-github/source-browser.html the repository]?
-Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages; here we assume XXX needs to be made from scratch. <br>
-ISO 639-3 codes can be found here: http://www-01.sil.org/iso639-3/codes.asp
-First make a new monolingual package:
-<pre>
 python3 apertium-init.py XXX
+# enter the directory
 cd apertium-XXX
+# compile the module
 make -j
-cd ..
 </pre>
+==Bootstrapping the pair==
-Then get and compile the existing monolingual package:
-<pre>
-apertium-get YYY
-</pre>
+In what follows, replace XXX and YYY for the [http://www-01.sil.org/iso639-3/codes.asp ISO 639-3] codes of your languages:
-Then generate the pair:
-<pre>
-python3 apertium-init.py XXX-YYY
-</pre>
-Then compile the pair:
-<pre>
-cd apertium-XXX-YYY
-./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
-make -j
-</pre>
-And test:
-<pre>
-echo house | apertium -d . XXX-YYY
-echo Haus | apertium -d . YYY-XXX
-</pre>
-Now you can add words to apertium-XXX-YYY.XXX-YYY.dix, then test again:
-<pre>
-make -j
-echo house | apertium -d . XXX-YYY
-echo Haus | apertium -d . YYY-XXX
-</pre>
-If you had to add words to the monolingual dictionaries, you will have to type "make" in those directories first. Alternatively, there is a shortcut from the pair directory: "make langs" should make the monolingual dictionaries even if you're in the pair directory.
-==With no existing monolingual packages==
-Do none of the two languagues you're making a pair of already have monolingual modules in [https://apertium.github.io/apertium-on-github/source-browser.html the repository]?
-Then follow this part, replacing XXX and YYY for the ISO 639-3 codes of your languages:
-First make and compile the new monolingual packages:
-<pre>
-python3 apertium-init.py XXX
-cd apertium-XXX
-make -j
-cd ..
-python3 apertium-init.py YYY
-cd apertium-YYY
-make -j
-cd ..
-</pre>
-Then generate the pair:
+Generate the pair:
 <pre>
 python3 apertium-init.py XXX-YYY

@@ Line 18: / Line 18: @@
 First compile the monolingual packages:
 <pre>
-git clone https://github.com/apertium/apertium-XXX.git
+apertium-get XXX
-cd apertium-XXX
+apertium-get YYY
-./autogen.sh
-make -j
-cd ..
-git clone https://github.com/apertium/apertium-YYY.git
-cd apertium-YYY
-./autogen.sh
-make -j
-cd ..
 </pre>
@@ Line 70: / Line 62: @@
 python3 apertium-init.py XXX
 cd apertium-XXX
-./autogen.sh
 make -j
 cd ..
@@ Line 77: / Line 68: @@
 Then get and compile the existing monolingual package:
 <pre>
-git clone https://github.com/apertium/apertium-YYY.git
+apertium-get YYY
-cd apertium-YYY
-./autogen.sh
-make -j
-cd ..
 </pre>
@@ Line 123: / Line 110: @@
 python3 apertium-init.py XXX
 cd apertium-XXX
-./autogen.sh
 make -j
 cd ..
@@ Line 129: / Line 115: @@
 python3 apertium-init.py YYY
 cd apertium-YYY
-./autogen.sh
 make -j
 cd ..
@@ Line 142: / Line 127: @@
 <pre>
+cd apertium-XXX-YYY
 ./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY
 make -j

@@ Line 172: / Line 172: @@
 See https://github.com/apertium/apertium-init for more documentation, or run <code>./apertium-init.py --help</code> for all options (you can e.g. also make pairs that don't use a statistical disambiguator, or don't use a Constraint Grammar disambiguator).
+== Choosing an Analyser ==
+Monolingual dictionaries can be set up to use 1 of 3 available formats: monodix, lexc, and lexd.
+Monodix is best for languages that have very little conjugation (such as English) or where the conjugation is entirely with suffixes but the stem doesn't change (such as Spanish). Monodix is the default setting for apertium-init or it can be explicitly specified with <code>--analyser=lttoolbox</code>.
+Lexc is best for languages where most of the conjugation is done with suffixes, potentially with some stem changes. Apertium-init will generate a Lexc dictionary with the option <code>--analyser=hfst</code>.
+Lexd should be used for any language with prefixes, infixes, circumfixes, or other morphology that isn't suffixes. Apertium-init will generate a Lexd dictionary with the option <code>--analyser=lexd</code>.
+If you're unsure which one to pick, Lexd will probably work well and generally involves less typing than either Monodix or Lexc.
 [[Category:Documentation]]

@@ Line 7: / Line 7: @@
 * apertium/lttoolbox/hfst, see [[Installation]], in particular the ''prerequisites'' parts. (You most likely don't need to go all the way, since you should get this stuff from Tino's repositories. If you're on Windows, get the [[Apertium VirtualBox]].)
-* [[apertium-init]].py – put this script in your working directory where you will be downloading language data (you only need https://raw.githubusercontent.com/apertium/apertium-init/master/apertium-init.py – not the whole repository –, although the information in the [https://github.com/apertium/apertium-init/blob/master/README.md README.md] file may be useful for troubleshooting)
+* [[apertium-init]].py – put this script in your working directory where you will be downloading language data (you only need https://apertium.projectjj.com/apt/nightly/apertium-init – not the whole repository –, although the information in the [https://github.com/apertium/apertium-init/blob/master/README.md README.md] file may be useful for troubleshooting)
 ==With two existing monolingual packages==