The quick and dirty guide to making a new language pair
Apertium New Language Pair HOWTO
This HOWTO document will describe how to start a new language pair for the Apertium machine translation system from scratch. You can check the list of language pairs that have already been started.
It does not assume any knowledge of linguistics, or machine translation above the level of being able to distinguish nouns from verbs (and prepositions etc.) You should probably familiarise yourself with basic Unix commands, though, if you're a complete newbie.
Apertium is, as you've probably realised by now, a machine translation system. Well, not quite, it's a machine translation platform. It provides an engine and toolbox that allow you to build your own machine translation systems. The only thing you need to do is write the data. The data consists, on a basic level, of three dictionaries and a few rules (to deal with word re-ordering and other grammatical stuff).
For a more detailed introduction into how it all works, there are some excellent papers on the Publications page.
These are the key steps one has to take in order to create a new language pair:
- Create the templates for a new language pair using the new bootstrap module.
- Create the morphological dictionary for the first language in the language pair (henceforth known as language xxx). This contains words in the first language for Apertium to recognise.
- Create the morphological dictionary for the second language in the language pair (henceforth known as language yyy). This contains words in the second language for Apertium to recognise.
- Create the bilingual dictionary for the desired language pair xxx-yyy. This contains correspondences between words and symbols in the two languages.
- Create the transfer rules for the language pair. This gives the Apertium rules to follow during translation of the language pair. When a pattern that correspond to these rules is detected in the source language text, Apertium will follow these transfer rules to create a grammatical translation in the target language.
- Follow Contributing to an existing pair for instructions on expansion and maintenance of the new language pair.
To make this HOWTO guide as straightforward and simple as possible, our example will create a fictional language pair xxx-yyy. We aim to guide you through the steps to create a basic working language pair, before you proceed to Contributing to an existing pair for the more advanced guide. We write this for the complete newbie to Apertium, but assume that you have basic knowledge of operating a Unix terminal.
Apertium is a rule-based machine translation platform that runs on Unix. If you're using Windows, you'll need to download a Virtual Machine. We have a ready-to-use Apertium VirtualBox for Windows users to quickly get started.
wget http://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get -f install apertium-all-dev
sudo yum install apertium-all-devel
sudo zypper install apertium-all-devel
# Mac OS X: sudo port install autoconf automake expat flex \ gettext gperf help2man libiconv libtool \ libxml2 libxslt m4 ncurses p5-locale-gettext \ pcre perl5 pkgconfig zlib gawk subversion
Before we start, we need to determine whether the languages you will be using are more suitable for lexc/twol (HFST) or lttoolbox based processing. "What on earth are those?", you ask. Bear me with me a bit.
- HFST is best for languages with agglutinative morphology and/or lots of morphophonology.
- lttoolbox is best for languages with fixed paradigms.
Languages with fairly limited morphology (e.g., English) can use either. Lttoolbox is the more default and common one, but if a language has agglutinative morphology, or any amount of productive morphophonology (especially vowel harmony), then it should use HFST.
If in doubt, please go to IRC and ask an expert.
For our example, we will imagine that xxx is a lttoolbox language, while yyy is a HFST language, so that I can demonstrate the addition of dictionaries for both types of languages.
Apertium has recently created a new bootstrap script to automatically create templates for new language pair creation. The older method of language-pair creation was, suffice it to say, more tedious.
- Get apertium-init from https://raw.githubusercontent.com/goavki/bootstrap/master/apertium-init.py
- Check the languages and incubator directories to find out if the languages that you will be working on are supported in Apertium. For our case, we will be starting from scratch, because our fictional "xxx" and "yyy" languages are not found in Apertium. Check the bootstrap wiki page for specific instructions if your languages can be found in Apertium. Also, remember to create the appropriate template depending if your language is lttoolbox or lexc/twol (HFST).
- Create the templates for language xxx.
- Create the template for language yyy.
- Create the template for language pair xxx-yyy.
To perform step 3 ("xxx" is an lttoolbox language):
python3 apertium-init.py xxx cd apertium-xxx ./autogen.sh make -j cd ..
To perform step 4 ("yyy" is a HFST language):
python3 apertium-init.py yyy --analyser=hfst cd apertium-yyy ./autogen.sh make -j cd ..
To perform step 5:
python3 apertium-init.py xxx-yyy --analyser2=hfst # Only yyy (second language) uses HFST cd apertium xxx-yyy ./autogen.sh --with-lang1=../apertium-XXX --with-lang2=../apertium-YYY make -j
Apertium uses dictionaries to identify words in a language. Only then can a word be processed by Apertium's tools.
Lttoolbox modules ("dix" files)
- See also: Dix
We shall begin by adding words to the dictionary of the first language "xxx". Enter "apertium-xxx" and open the dix file (it should be named "apertium-xxx.xxx.dix"). You will see a few lines:
<?xml version="1.0" encoding="UTF-8"?> <dictionary> </dictionary> <alphabet>ABCČĆDDžĐEFGHIJKLLjMNNjOPRSŠTUVZŽabcčćddžđefghijklljmnnjoprsštuvzž</alphabet>
Edit the alphabet to be your appropriate for your language.
Next, you will see some symbol definitions:
<sdefs> <sdef n="n"/> <sdef n="sg"/> <sdef n="pl"/> </sdefs>
The template is very intuitive. "n" means noun, "sg" means singular, and "pl" means plural. Do, however, be familiar with Apertium's standard List of symbols. Do be also prepared to add more symbols if your language demands it. For example, nouns in Serbo-Croatian inflect for more than just number, they are also inflected for case, and have a gender. Appropriate symbols will have to be added to the language dictionary to reflect these language features.
The next thing you will see is the section for paradigms. Paradigms are 'structures' so to say, that define the part of speech traits (such as noun, verb, transitive, etc) of a certain word. Then, any further words in the dictionary that behave similarly to the given word can be assigned this paradigm, to mark that they have the same traits and can be treated similarly. It's a faster way to assign part-of-speech traits to words without typing stuff over and over again. You'll be adding more paradigms in future when expanding the language (and hence, language pair), as many as different word forms found in the language itself.
Add the most basic of singular nouns for now. In English, this can be "house".
<pardef n="house__n"> <e> <p><l></l> <r><s n="n"/><s n="sg"/></r></p></e> </pardef>
Add the e, p and l tags, then add r tags and fit the appropriate defining symbols inside it. <l> tags include different inflections of the word (but the word's base form essentially remains the same). So, the basic word "house" can actually have four paradigm entries.
<pardef n="house__n"> <e> <p><l></l> <r><s n="n"/><s n="sg"/></r></p></e> <e> <p><l>'s</l> <r><s n="n"/><s n="sg"/><s n="gen"/></r></p></e> <e> <p><l>s</l> <r><s n="n"/><s n="pl"/></r></p></e> <e> <p><l>s'</l> <r><s n="n"/><s n="pl"/><s n="gen"/></r></p></e> </pardef>
You do not have to concern yourself with the specifics of what the tags <e> and <'p'> do for now.
Let's now add words to the dictionary. Nouns are the easiest to begin with for most languages. It may also be wise to look at the dix files of other languages to personally observe the format for yourself.
<e lm="abbot"><i>abbot</i><par n="house__n"/></e>
The format is: <e lm="wordhere">wordhere<the most appropriate paradigm here></e>
Once you're done, save it and compile the folder again with
If your second language also uses a dix file, do the same thing, substituting the noun you added to xxx's dictionary with the appropriate translation in yyy.
HFST modules (lexc/twol files)
- See also: Starting a new language with HFST
HFST-type languages are more difficult to begin, because of the nature of their more complex morphology.
Now, for the dictionary of the other language pair ("yyy"), do the same thing, substituting the noun you added to xxx's dictionary with the appropriate translation in yyy.
You should now have two files:
- apertium-xxx.xxx.dix which contains a (very) basic xxx morphological dictionary, and
- apertium-yyy.yyy.dix which contains a (very) basic yyy morphological dictionary.
See also: Bilingual dictionary
So we now have two morphological dictionaries, next thing to make is the bilingual dictionary. This describes mappings between words. During translation, one word will be translated to its corresponding word as specified in this dictionary.
Add an entry to translate between the two words, as specified in the monolingual dictionaries. Something like:
<e><p><l>xxx<s n="n"/></l><r>yyy<s n="n"/></r></p></e>
There are going to be a lot of these entries, so it would be wise to write every entry on one line to facilitate easier reading of the file.
So, once this is done, run
make -j to compile the bilingual dictionary.
You're done with the addition of dictionaries!
Testing of dictionaries
This is a good checkpoint to test if your language pair works, and that you have set up everything correctly up till this point.
Make sure that you are in the folder with the language pair, and type an echo command:
echo xxx | apertium -d . xxx-yyy echo yyy | apertium -d . yyy-xxx
If this works, great! We move on to the next section: The addition of transfer rules.
See also: Transfer
So, now we have two morphological dictionaries, and a bilingual dictionary. All that we need now is a transfer rule for nouns. Transfer rules are necessary in sentence-based translation to translate longer chunks of text grammatically. They also help assign the correct tags for the part-of-speech tagger that is necessary in the development of a language pair.
If you need to implement a rule it is often a good idea to look in the rule files of other language pairs first. Many rules can be recycled/reused between languages.
Add a rule to take in the nouns and then output it in the correct form.
I will use the example of a language pair English-Malay (eng-zlm) to demonstrate a simple transfer rule. In English, the adjective comes before the noun, such as "happy boy". However, in Malay, the noun comes before the adjective, such as "lelaki gembira" (boy happy). Remember to modify the rule as is appropriate for your language.
Note: You need to add the words that you will want test in the dictionaries first, before you can test them!
<rule comment="REGLA: adj nom"> <!--happy boy--> <pattern> <pattern-item n="adj"/> <pattern-item n="nom"/> </pattern> <action> <out> <chunk name="j_n" case="caseFirstWord"> <lu> <clip pos="2"/> <!--pos 2 refers to the second word in the rule ("boy"). It is now placed at the front.--> </lu> <b/> <!-- this is the space to indicate the word should be separated--> <lu> <clip pos="1"/> <!--pos 1 refers to the first word in the rule ("happy"). It is now placed at the back.--> </lu> </chunk> </out> </action> </rule>
For each pattern, there is an associated action, which produces an associated output, out. The output, is a lexical unit (lu).
The clip tag allows a user to select and manipulate attributes and parts of the source language (side="sl"), or target language (side="tl") lexical item. This can get very complicated, so let's move on. Further elaboration can be found in A long introduction to transfer rules.
Let's compile it and test it.
Now we're ready to test our machine translation system.
$ echo "happy boy" | apertium -d . eng-zlm ^lelaki/lelaki<n><sg>$ ^gembira/gembira<adj>$
And c'est ca. You now have a machine translation system that translates words between your chosen two languages. You have also learned how to write a basic transfer rule, that grammatically translates longer chunks of words into their target language. Obviously this isn't very useful, but you'll get onto the more complex stuff soon.
Think of a few other words that inflect the same as the nouns in your dictionaries, in our case, "xxx" and "yyy". How about adding those? You can link their entries with the paradigm you had written earlier.
See also: Contributing to an existing pair
Congratulations on creating your new infant language pair! Your journey is (obviously) not over yet, though! There is a long way to go before the language pair becomes stable. Steps include: adding more words to the dictionaries, adding more transfer rules to accommodate them, training the part-of-speech tagger, bug and error testing, and more.
As is the usual, feel free to browse the wiki for more resources, or proceed to IRC for help.
We wish you the best of luck in the creation of your new language pair!