Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Become a language pair developer for Apertium

From Apertium
(Difference between revisions)
Jump to: navigation, search
 
(29 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{TOCD}}This is a 3-part, step-by-step guide on how to use a development version of Apertium to make a change in a language pair. These instructions assume that you are using Ubuntu or Debian; if not then please see the [[Installation]] page for instructions for other OS's such as Mac OS X or Windows.
+
[[Devenir un développeur de paire de langues pour Apertium|En français]]
   
== Intro ==
+
{{TOCD}}
  +
{{Github-migration-check}}
  +
This is a step-by-step guide on how to use a development version of Apertium to make a change in a language pair.
   
There are 2 options as to how you can get Apertium. You can use either the Terminal to get the most up-to-date versions or the Synaptic Package Manager can be used to get development versions that aren't quite as up-to-date. There are pros and cons to each, however, the Terminal method is more for those that intend to submit their work while using the package manager is normally easier and you are using a graphical interface instead of a command line.
+
== Introduction ==
   
'''YOU DON'T NEED TO KNOW A PROGRAMMING LANGUAGE TO DEVELOP FOR APERTIUM. All development for adding new words is done with a text editor.'''
+
You will need:
+
* a text editor, preferably with good XML highlighting – see [[XML editors]] for some popular choices,
  +
* a thorough understanding of the languages you wish to develop for,
  +
* an [[IRC]] client so you can ask for help when you get into trouble,
  +
  +
If you wish to contribute to the main repository, you will also have to ask for commit access on [http://sourceforge.net/projects/apertium/ SourceForge]
   
 
== Getting Ready ==
 
== Getting Ready ==
  +
The first thing you need to do is install the Apertium core modules. The page [[Installation]] shows how to do this, usually by packaging (non-Apple Unix based operating systems), or a virtual environment (Apple/Windows operating systems).
  +
  +
The [[Install language data by compiling]] page shows how to install an example language pair, e.g. apertium-eo-enor apertium en-ca, both of which are known and stable. You should probably try this to make sure things work before you move on to whatever language pair you plan on working on.
   
=== Method 1: TERMINAL ===
+
Note that some existing language pairs have external dependencies, like HFST or Constraint Grammar. The [[Installation]] page links to their installation procedures (if using packaging or a virtual environment, they are either a one-click install, or pre-installed).
   
==== Step 1: Get the Prerequisites ====
 
   
A development version of Apertium and the language pair you want to change has to be installed on your computer first before you can change something about the language pair.
 
   
Start by opening a new Terminal.
+
==== Get your language pair(s) ====
   
Then, use this command to install the prerequisites:
+
Using the same terminal, you can easily download and add the language pairs you want using a command like:
 
<pre>
 
<pre>
sudo apt-get install subversion build-essential g++ pkg-config gawk libxml2 \
+
svn checkout svn://svn.code.sf.net/p/apertium/svn/SVN_MODULE/PAIR_NAME
> libxml2-dev libxml2-utils xsltproc flex automake autoconf libtool libpcre3-dev
 
 
</pre>
 
</pre>
   
The Terminal will then ask you for your password to begin.
+
In the area where it says SVN_MODULE, replace this with the the name of the svn subdirectory where the chosen language pair is.
   
''Note: Keep track of how you type your password in your head. The Terminal will not display characters entered for your password for security reasons.''
+
In the area where it says PAIR_NAME, replace this with the name of the chosen language pair.
   
After you have entered your password, press the "Enter" key and wait for your computer to download and install the packages.
+
For example, if you wanted to retrieve the language pair Spanish/English (which is in trunk) and French/Portuguese (which was in staging as of June 2012) you could type:
 
==== Step 2: Get Apertium, lttoolbox, & Your Language Pair(s) ====
 
 
Using the same Terminal, you can download the entire language pairs tree from SVN using the command:
 
<pre>
 
svn co https://apertium.svn.sourceforge.net/svnroot/apertium
 
</pre>
 
Keep in mind that the full tree is over 4GB. If you have a slow connection, limited disk space, or a limited data transfer amount, installing the whole tree is not recommended. However, if you want to focus on a small number of language pairs you can easily download and add language pairs with a command such as:
 
 
<pre>
 
<pre>
svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/<modulename>
+
svn checkout svn://svn.code.sf.net/p/apertium/svn/trunk/apertium-en-es
  +
svn checkout svn://svn.code.sf.net/p/apertium/svn/staging/apertium-fr-pt
 
</pre>
 
</pre>
''Note: This command only downloads one module at a time. For a more complete set of instructions on effectively using SVN, see the [[Using SVN]] page.''
 
   
  +
You can go to [https://apertium.svn.sourceforge.net/svnroot/apertium/] to see a list of SVN modules, clicking on each shows a list of language pairs. Language pairs are sorted into SVN modules based on how "complete" they are, where [[trunk]] is release-quality, [[staging]] is very close to release-quality, [[nursery]] is for stuff that's 1-3 months of concentrated work from release quality, and [[incubator]] is fragments and anything not complete enough to live in the other modules.
   
In the area where it says <modulename>, you can replace this with the module that you want to use such as the Spanish/English module.
 
   
These next commands download Apertium, lttoolbox, and the language pair that you want to use.
 
<pre>
 
svn co http://apertium.svn.sourceforge.net/svnroot/apertium/trunk/lttoolbox
 
svn co http://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium
 
svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/<modulename>
 
</pre>
 
   
  +
==== Compilation of your language pair ====
  +
  +
Compilation of a language pair is similar to lttoolbox and apertium, except you don't need to do "sudo ldconfig", and you should ''not'' do "sudo make install" since it's easier and faster to run from the source folder:
   
For example, if you wanted to get Apertium, lttoolbox, and the Spanish/English module you could enter:
 
 
<pre>
 
<pre>
svn co http://apertium.svn.sourceforge.net/svnroot/apertium/trunk/lttoolbox
+
cd PAIR_NAME
svn co http://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium
+
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh
svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium-en-es
+
make -j3
 
</pre>
 
</pre>
   
''Note: You can find a full list of modules at [https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/ https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/].''
+
for each language pair, replacing the text PAIR_NAME by the appropriate name.
   
==== Final Step: Compilation & Installation ====
+
== Changing Things Around ==
   
First, you need to compile lttoolbox, Apertium, and your language pair and install them. For this we will use,
+
When you want to make a change in Apertium, you more than likely want to add a word to an existing language pair. For a full explanation go to [[Contributing to an existing pair]]. You can also check out the [[Contact]] page for Apertium mailing lists and live help through IRC.
  +
  +
'''IMPORTANT: Adding a word won't do you any good if you don't recompile the modules after the change is made. Simply use the terminal like before and enter:''' ''make'' ''' and press the "Enter" key and your computer will create the new files necessary.'''
  +
  +
  +
There are 3 major steps in adding a new word to a language pair:
  +
  +
'''1.''' Add an entry to the dictionary for the first language that will be used.
  +
  +
'''2.''' Add an entry to the bilingual dictionary for the pair.
  +
  +
'''3.''' Add an entry to the dictionary for the second language that will be used.
  +
  +
You will need to find the module you want to work with on your computer and open the three dictionaries; for example: apertium-es-ca.es.dix,
  +
apertium-es-ca.es-ca.dix, and apertium-es-ca.ca.dix. ''Note: Each dictionary will have the suffix ".dix"'' You should open these files in a text editor or specialized XML editor.
  +
  +
=== Step 1: Adding to the First Dictionary ===
  +
  +
When adding entries, you have to enter the lemma (word as you would read it in a dictionary),the part between <nowiki><i></nowiki> and <nowiki></i></nowiki> that contains the prefix of the word that is common to all inflected forms, and the element in <par> that refers to the inflection paradigm of this word. All entries will have a basic structure like:
 
<pre>
 
<pre>
cd apertium
+
<e lm="(lemma)">
cd lttoolbox/
+
<i>(prefix)</i>
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh
+
<par n="(paradigm)"/>
make
+
</e>
sudo make install
 
sudo ldconfig
 
 
</pre>
 
</pre>
, for lttoolbox. Then,
+
A good example of this would be:
 
<pre>
 
<pre>
cd ..
+
<e lm="cósmico">
cd apertium/
+
<i>cósmic</i>
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh
+
<par n="absolut/o__adj"/>
make
+
</e>
sudo make install
 
sudo ldconfig
 
 
</pre>
 
</pre>
for Apertium. And finally,
+
  +
Start by opening your first language's dictionary file. For example: apertium-en-es.es.dix (an XML file).
  +
  +
Then, create a new entry with the basic structure next to a similar entry in the dictionary. The order of entries doesn't matter.
  +
  +
Now, between the quotes in the area where it says (lemma) replace (lemma) with your word. ''Note: Do not include () in entries, but place input between "".''
  +
  +
Next, you can enter the prefix in the space between <nowiki><i></nowiki> and <nowiki></i></nowiki> and replace (prefix).
  +
  +
Finally, enter the paradigm in <par> between the quotations. The paradigm should consist of the prefix of another word that has the same inflection and is already in the dictionary and has the morphological analysis: adj m sg, adj f sg, adj m pl and adj f pl respectively. For example: <par n="absolut/o__adj"/> for cósmico. This entry means that the adjective "cósmico" inflects like the adjective "absoluto" and has the same morphological analysis: the forms cósmico, cósmica, cósmicos, and cósmicas are equivalent to the forms absoluto, absoluta, absolutos, and absolutas and have the morphological analysis: adj m sg, adj f sg, adj m pl and adj f pl respectively.
  +
  +
Now, save your altered dictionary, and '''DO NOT''' change file name, directory, or file type.
  +
  +
To finish, use the terminal and navigate to the directory that your module is housed in and enter ''make''. Now press the "Enter" key and allow you computer to recompile the module with the changes you just made.
  +
  +
=== Step 2: Adding to the Second Dictionary ===
  +
  +
Using the same structure, you can create the entry in your second language's dictionary that is the equivalent to your entry in the first dictionary.
  +
  +
The second language dictionary file name should be something such as ''apertium-en-es.en.dix''.
  +
  +
Save your changes and recompile the module.
  +
  +
=== Final Step: The Bilingual Dictionary ===
  +
  +
Adding entries to the bilingual dictionary is considerably easier than adding to the other two dictionaries. An entry in this dictionary has this basic structure:
 
<pre>
 
<pre>
cd ..
+
<e>
cd <modulename>/
+
<p>
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh
+
<l>(lemmafromfirst)<s n="(partofspeech)"/></l>
make
+
<r>(lemmafromsecond)<s n="(partofspeech)"/></r>
sudo make install
+
</p>
  +
</e>
 
</pre>
 
</pre>
,for your language pair which replaces the text <modulename>.
 
   
For further instruction, if necessary, see [[Apertium on Ubuntu]].
+
Simply add an entry and replace (lemmafromfirst) with the lemma you added to the first dictionary, (lemmafromsecond) with the lemma from the second, and (partofspeech) with the part of speech for each word.
   
=== Method 2: PACKAGE MANAGER ===
+
Save this dictionary and recompile the module one last time.
   
Using the Synaptic Package Manager to download and install Apertium, lttoolbox, and your language pair is considerably easier than the Terminal method, however, your choice of language pairs is limited, you may be unable to commit changes, and there may possibly be other flaws or minor bugs.
+
Adding rules to a language pair can also be done, however, that will not be discussed in this guide. See [[Contributing to an existing pair]] for a more detailed and full explanation.
   
==== Step 1: Find your Packages ====
+
=== Errors ===
   
To begin, start by finding the Synaptic Package Manager and opening it.
+
It is very possible that you may encounter an error in you changes.
   
Then, use the search box and type in "apertium".
+
To know how a word is analysed by the translator and find an error, type the following in the terminal (example from [[Contributing_to_an_existing_pair#Detecting_errors|Contributing to an existing pair. Follow link for more help.]]):
  +
<pre>
  +
$ echo "gener" | apertium-destxt | lt-proc ca-es.automorf.bin
  +
</pre>
   
Synaptic should bring up a list of everything related to Apertium. This list should include language pairs, the development versions of lttoolbox and libapertium, as well as the Apertium base package and various others.
+
You can replace ca-es with the translation direction you want to test.
   
==== Step 2: Compilation & Installation ====
+
The output in Apertium should be:
  +
<pre>
  +
^gener/gener<n><m><sg>$^./.<sent>$[][]
  +
</pre>
  +
The string structure is: <code>^word/lemma<morphological analysis>$</code>. The <code><sent></code> tag is the analysis of the full stop, as every sentence end is represented as a full stop by the system, whether or not explicitly indicated in the sentence.
   
Luckily, Synaptic takes care of getting the prerequisites, dependencies, and other required packages.
+
The analysis of an unknown word is (ignoring the full stop information):
== Changing Things Around ==
 
   
When you want to make a change in Apertium, you more than likely want to add a word to an existing language pair.
+
<pre>
  +
^genoma/*genoma$
  +
</pre>
   
'''IMPORTANT: Adding a word won't do you any good if you don't recompile the modules after the change is made. Simply use the Terminal like before and enter:''' ''make <modulenamehere>'' '''and press the "Enter" key and your computer will create the new files necessary.'''
+
and the analysis of an ambiguous word:
   
  +
<pre>
  +
^casa/casa<n><f><sg>/casar<vblex><pri><p3><sg>/casar<vblex><imp><p2><sg>$
  +
</pre>
   
There are 3 major steps in adding a new word to a language pair:
+
Each lexical form (lemma plus morphological analysis) is presented as a possible analysis of the word casa.
   
'''1.''' Add an entry to the dictionary for the first language that will be used.
+
If you are still stuck remember that you can always ask questions on [[IRC]].
   
'''2.''' Add an entry to the bilingual dictionary for the pair.
+
== Show it to the World ==
   
'''3.''' Add an entry to the dictionary for the second language that will be used.
+
Now that you have added to a language pair you have the option to commit your changes to SourceForge. Committing a change to a language pair is even easier than making the change.
   
You will need to find the module you want to work with on your computer and open the three dictionaries; for example: apertium-es-ca.es.dix,
+
Firstly, you need to register for a free SourceForge account. Then, contact an Apertium administrator [http://sourceforge.net/project/memberlist.php?group_id=143781 here] and request access to commit to SVN on SourceForge.
apertium-es-ca.es-ca.dix, and apertium-es-ca.ca.dix. ''Note: Each dictionary will have the suffix ".dix"'' You should open these files in a text editor or specialized XML editor.
 
   
'''Step 1: Adding to the First Dictionary'''
+
Once granted access, simply open the terminal, navigate to your language pair that was changed, and enter:
  +
<pre>
  +
svn commit
  +
</pre>
   
'''IN PROGRESS'''
+
Remember to include a log message detailing what was changed or added.
   
== Show it to the World ==
+
It is also possible that ''svn commit'' may be followed by -m "message”. That is sometimes easier than having an editor opened automatically.
  +
  +
You have now become a language pair developer for Apertium!
  +
  +
==See also==
  +
  +
*[[Building dictionaries]]
  +
*[[Contributing to an existing pair]]
  +
*[[Finding_errors_in_dictionaries]]
  +
*[[Apertium_New_Language_Pair_HOWTO]]
  +
*[[Cookbook]]
  +
*[[Chunking]]
   
'''IN PROGRESS'''
+
[[Category:Documentation in English]]
  +
[[Category:Documentation]]
  +
[[Category:HOWTO]]
  +
[[Category:Writing dictionaries]]
  +
[[Category:Quickstart]]

Latest revision as of 21:04, 22 January 2019

En français

Contents

WARNING

This page is out of date as a result of the migration to GitHub. Please update this page with new documentation and remove this warning. If you are unsure how to proceed, please contact the GitHub migration team.

This is a step-by-step guide on how to use a development version of Apertium to make a change in a language pair.

[edit] Introduction

You will need:

  • a text editor, preferably with good XML highlighting – see XML editors for some popular choices,
  • a thorough understanding of the languages you wish to develop for,
  • an IRC client so you can ask for help when you get into trouble,

If you wish to contribute to the main repository, you will also have to ask for commit access on SourceForge

[edit] Getting Ready

The first thing you need to do is install the Apertium core modules. The page Installation shows how to do this, usually by packaging (non-Apple Unix based operating systems), or a virtual environment (Apple/Windows operating systems).

The Install language data by compiling page shows how to install an example language pair, e.g. apertium-eo-enor apertium en-ca, both of which are known and stable. You should probably try this to make sure things work before you move on to whatever language pair you plan on working on.

Note that some existing language pairs have external dependencies, like HFST or Constraint Grammar. The Installation page links to their installation procedures (if using packaging or a virtual environment, they are either a one-click install, or pre-installed).


[edit] Get your language pair(s)

Using the same terminal, you can easily download and add the language pairs you want using a command like:

svn checkout svn://svn.code.sf.net/p/apertium/svn/SVN_MODULE/PAIR_NAME

In the area where it says SVN_MODULE, replace this with the the name of the svn subdirectory where the chosen language pair is.

In the area where it says PAIR_NAME, replace this with the name of the chosen language pair.

For example, if you wanted to retrieve the language pair Spanish/English (which is in trunk) and French/Portuguese (which was in staging as of June 2012) you could type:

svn checkout svn://svn.code.sf.net/p/apertium/svn/trunk/apertium-en-es
svn checkout svn://svn.code.sf.net/p/apertium/svn/staging/apertium-fr-pt

You can go to [1] to see a list of SVN modules, clicking on each shows a list of language pairs. Language pairs are sorted into SVN modules based on how "complete" they are, where trunk is release-quality, staging is very close to release-quality, nursery is for stuff that's 1-3 months of concentrated work from release quality, and incubator is fragments and anything not complete enough to live in the other modules.


[edit] Compilation of your language pair

Compilation of a language pair is similar to lttoolbox and apertium, except you don't need to do "sudo ldconfig", and you should not do "sudo make install" since it's easier and faster to run from the source folder:

cd PAIR_NAME
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh
make -j3

for each language pair, replacing the text PAIR_NAME by the appropriate name.

[edit] Changing Things Around

When you want to make a change in Apertium, you more than likely want to add a word to an existing language pair. For a full explanation go to Contributing to an existing pair. You can also check out the Contact page for Apertium mailing lists and live help through IRC.

IMPORTANT: Adding a word won't do you any good if you don't recompile the modules after the change is made. Simply use the terminal like before and enter: make and press the "Enter" key and your computer will create the new files necessary.


There are 3 major steps in adding a new word to a language pair:

1. Add an entry to the dictionary for the first language that will be used.

2. Add an entry to the bilingual dictionary for the pair.

3. Add an entry to the dictionary for the second language that will be used.

You will need to find the module you want to work with on your computer and open the three dictionaries; for example: apertium-es-ca.es.dix, apertium-es-ca.es-ca.dix, and apertium-es-ca.ca.dix. Note: Each dictionary will have the suffix ".dix" You should open these files in a text editor or specialized XML editor.

[edit] Step 1: Adding to the First Dictionary

When adding entries, you have to enter the lemma (word as you would read it in a dictionary),the part between <i> and </i> that contains the prefix of the word that is common to all inflected forms, and the element in <par> that refers to the inflection paradigm of this word. All entries will have a basic structure like:

      <e lm="(lemma)">
        <i>(prefix)</i>
        <par n="(paradigm)"/>
      </e>

A good example of this would be:

      <e lm="cósmico">
        <i>cósmic</i>
        <par n="absolut/o__adj"/>
      </e>

Start by opening your first language's dictionary file. For example: apertium-en-es.es.dix (an XML file).

Then, create a new entry with the basic structure next to a similar entry in the dictionary. The order of entries doesn't matter.

Now, between the quotes in the area where it says (lemma) replace (lemma) with your word. Note: Do not include () in entries, but place input between "".

Next, you can enter the prefix in the space between <i> and </i> and replace (prefix).

Finally, enter the paradigm in <par> between the quotations. The paradigm should consist of the prefix of another word that has the same inflection and is already in the dictionary and has the morphological analysis: adj m sg, adj f sg, adj m pl and adj f pl respectively. For example: <par n="absolut/o__adj"/> for cósmico. This entry means that the adjective "cósmico" inflects like the adjective "absoluto" and has the same morphological analysis: the forms cósmico, cósmica, cósmicos, and cósmicas are equivalent to the forms absoluto, absoluta, absolutos, and absolutas and have the morphological analysis: adj m sg, adj f sg, adj m pl and adj f pl respectively.

Now, save your altered dictionary, and DO NOT change file name, directory, or file type.

To finish, use the terminal and navigate to the directory that your module is housed in and enter make. Now press the "Enter" key and allow you computer to recompile the module with the changes you just made.

[edit] Step 2: Adding to the Second Dictionary

Using the same structure, you can create the entry in your second language's dictionary that is the equivalent to your entry in the first dictionary.

The second language dictionary file name should be something such as apertium-en-es.en.dix.

Save your changes and recompile the module.

[edit] Final Step: The Bilingual Dictionary

Adding entries to the bilingual dictionary is considerably easier than adding to the other two dictionaries. An entry in this dictionary has this basic structure:

     <e> 
        <p>
          <l>(lemmafromfirst)<s n="(partofspeech)"/></l>
          <r>(lemmafromsecond)<s n="(partofspeech)"/></r>
        </p>
      </e>

Simply add an entry and replace (lemmafromfirst) with the lemma you added to the first dictionary, (lemmafromsecond) with the lemma from the second, and (partofspeech) with the part of speech for each word.

Save this dictionary and recompile the module one last time.

Adding rules to a language pair can also be done, however, that will not be discussed in this guide. See Contributing to an existing pair for a more detailed and full explanation.

[edit] Errors

It is very possible that you may encounter an error in you changes.

To know how a word is analysed by the translator and find an error, type the following in the terminal (example from Contributing to an existing pair. Follow link for more help.):

$ echo "gener" | apertium-destxt | lt-proc ca-es.automorf.bin

You can replace ca-es with the translation direction you want to test.

The output in Apertium should be:

^gener/gener<n><m><sg>$^./.<sent>$[][]

The string structure is: ^word/lemma<morphological analysis>$. The <sent> tag is the analysis of the full stop, as every sentence end is represented as a full stop by the system, whether or not explicitly indicated in the sentence.

The analysis of an unknown word is (ignoring the full stop information):

^genoma/*genoma$

and the analysis of an ambiguous word:

^casa/casa<n><f><sg>/casar<vblex><pri><p3><sg>/casar<vblex><imp><p2><sg>$

Each lexical form (lemma plus morphological analysis) is presented as a possible analysis of the word casa.

If you are still stuck remember that you can always ask questions on IRC.

[edit] Show it to the World

Now that you have added to a language pair you have the option to commit your changes to SourceForge. Committing a change to a language pair is even easier than making the change.

Firstly, you need to register for a free SourceForge account. Then, contact an Apertium administrator here and request access to commit to SVN on SourceForge.

Once granted access, simply open the terminal, navigate to your language pair that was changed, and enter:

svn commit

Remember to include a log message detailing what was changed or added.

It is also possible that svn commit may be followed by -m "message”. That is sometimes easier than having an editor opened automatically.

You have now become a language pair developer for Apertium!

[edit] See also

Personal tools