Lextor

When the Apertium system is used to translate between less related languages than the ones dealt with in the first stages of the engine, the question of lexical selection becomes significant, because there are more cases, and more critical, in which a source language word can have more than one different translation in the target language. For this reason we created a new module, the lexical selection module, which deals with this problem.

Before going into its characteristics, we will see how the problems of multiple equivalence (the fact of existing more than one possible translation in target language for a source language lexical form) are tackled in Apertium in two ways.

On the one hand, we have the situation where there is no big difference in meaning between the multiple equivalents in the target language, and the fact of choosing one or the other can not lead to any translation error. We could say that between these equivalents there is a synonymy or quasi-synonymy relation. In such a case, the linguist chooses one of the lemmas as a translation (generally the most frequent or usual), and adds a direction restriction to the other lemmas (with the attributes LR or RL) so that they are translated in the opposite direction but not in the direction where there are multiple equivalents.

On the other hand, we have the case where there is a clear difference in meaning between the multiple equivalents, which can lead to translation errors if the inappropriate lemma is chosen. These are the cases dealt with the new lexical selection module. The linguist has to encode entries with the attributes slr or srl described in the next section, thus identifying the different translation options; then, the lexical selection module, by means of statistical methods, chooses the translation which is most suitable in a given context.

Sometimes it is not easy to decide whether a multiple equivalence situation should be solved in one way or the other. For example, if there is difference in the meaning of two or more lemmas in the target language, but we think that the lexical selection module will not be capable of choosing the right translation by means of the context, we will follow the first method: choose a fixed translation (the most general, the most suitable in the maximum number of situations) and add a direction restriction to the rest of translations. In the other cases, we will encode the entries so that the decision is left to the lexical selection module.

When we use an Apertium system without lexical selection module, the only way to add entries with different possible translations is the first one, that is, choosing an only translation and marking the other equivalences with a direction restriction. In the event that we use bilingual dictionaries with multiple translations, encoded with the attributes slr or srl, in a system that does not have any lexical selection module, a style sheet will convert these entries designed for a lexical selection module into entries with direction restrictions LR or RL, so that one of the multiple equivalents (the one chosen as default entry by the linguist) becomes the fixed translation of the source language lemma.

As examples of bilingual equivalencies that should have a direction restriction, we can give the translation pairs ca-es encara — aún or todavía ("still") and sobtat — súbito or repentino ("sudden"), the first one of which could be encoded like this:

<e r="LR">
   <p>
      <l>aún<s n="adv"/></l>
      <r>encara<s n="adv"/></r>
   </p>
</e>
<e>
    <p>
      <l>todavía<s n="adv"/></l>
      <r>encara<s n="adv"/></r>
    </p>
</e>

As examples of the second case (multiple equivalents with big difference in meaning) we have the pairs es-ca hoja — full or fulla ("sheet/leaf") and muñeca — nina or canell ("doll/wrist"), as well as the en-ca examples shown in page X, where it is described how to specify these multiple equivalents in the bilingual dictionary.

\begin{figure} {\footnotesize \setlength{\tabcolsep}{0.5mm} \begin{center} \begin{tabular}{ccccccccc} \\ \parbox{0.95cm}{source language text} \\ $\downarrow$ \\ \framebox{\parbox{1.0cm}{de-for\-matter}} $\rightarrow$ & \framebox{\parbox{0.6cm}{morph. anal.}} $\rightarrow$ & \framebox{\parbox{1.0cm}{POS tagger}} $\rightarrow$ & \framebox{\parbox{0.6cm}{lex. select.}} $\rightarrow$ & \framebox{\parbox{0.85cm}{struct. transf.}} $\rightarrow$ & \framebox{\parbox{0.6cm}{morph. gen.}} $\rightarrow$ & \framebox{\parbox{1.2cm}{post\-generator}} $\rightarrow$ & \framebox{\parbox{1.0cm}{re-for\-matter}} \\ & & & & $\updownarrow$ & & & $\downarrow$ \\ & & & & \framebox{\parbox{0.8cm}{lex. transf.}} & & & \parbox{0.95cm}{target language text} \\ \end{tabular} \end{center} } \caption{The nine modules that build the assembly line in the version

 2 of the machine translation system Apertium.}

\label{fig:moduls} \end{figure}

Figure~\ref{fig:moduls} shows the new assembly line of the version 2 of Apertium.\footnote{This figure substitutes the figure \ref{fg:modules} in page \pageref{pg:modules} which represents the version 1 of Apertium.} \nota{MG: caldria canviar la figura de la pàgina 6 per aquesta d'aquí?} The module in charge of the lexical selection (lexical selector) runs after the part-of-speech tagger and before the structural transfer module; therefore, this new module works only with source language information.

Section~\ref{se:preprocessament} next describes the pre-processing that must be done on a bilingual dictionary containing more than one translation per entry (whether the system uses a lexical selector or not), and Section~\ref{se:lextor} describes how the lexical selector works and how it has to be trained.

Pre-processing of the bilingual dictionaries

Bilingual dictionaries have been modified to allow the specification of more than one translation per entry (refer to Section \ref{dic_lextor} to learn how to write such dictionary entries); this fact makes it necessary to pre-process these dictionaries, since the Apertium engine works with compiled dictionaries in which there is only one possible translation for each word.

The pre-processing of dictionaries is done automatically during compilation, therefore the final user does not need to perform any specific action.

Pre-processing without lexical selection module

When bilingual dictionaries with multiple equivalents are used in a system where there is no lexical selection module, the pre-processing is done by the application of the style sheet translate-to-default-equivalent.xsl. This style sheet turns dictionaries with multiple translations per entry into dictionaries with only one translation per entry; to do this, it chooses as translation the entry marked as default, and adds a direction restriction (LR or RL as applicable) to the other entries, so that they are only translated in the translation direction where there is no equivalent multiplicity. The style sheet is called from the Makefile.

To put an example, the result of applying the style sheet on the first three entries shown in page \pageref{entrades_lextor} is the following:

<e>
   <p>
      <l>flat<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
   </p>
</e>

<e r="LR">
   <p>
      <l>floor<s n="n"/></l>
      <r>pis<s n="n"/><s n="m"/></r>
   </p>
</e>

<e r="RL">
   <p>
      <l>floor<s n="n"/></l>
      <r>terra<s n="n"/><s n="m"/></r>
   </p>
</e>

Preprocessing with lexical selection module

If the Apertium system works with a lexical selection module, the bilingual dictionary must be pre-processed in order to obtain:

• a monolingual dictionary that, for each source language word

(for example \emph{look}) delivers all the possible translation marks or equivalents (look\_\_mirar D} and look\_\_semblar}); this dictionary will be used by the lexical selection module; and

• a new bilingual dictionary that, given a word with the lexical

selection already done (for example look\_\_semblar}) delivers the translation (\emph{semblar}); this will be the bilingual dictionary to be used in the lexical transfer.

This pre-processing is automatically done by means of the following software during dictionary compilation:

• apertium-gen-lextormono, that receives three parameters:

1. the translation direction for which you want to generate the monolingual dictionary used in the lexical selection; lr for the translation left to right, and rl for the translation right to left;
2. the monolingual dictionary to be pre-processed; and
3. the file where the output monolingual dictionary has to be written.

• apertium-gen-lextorbil, that receives three parameters:

1. the translation direction (lr or rl) for which you want to generate the bilingual dictionary to be used by the lexical transfer module;
2. the bilingual dictionary to be pre-processed; and
3. the file where the output bilingual dictionary has to be written.

Execution of the lexical selection module

The module responsible for the lexical selection runs after the part-of-speech tagger and before the structural transfer (see Figure~\ref{fig:moduls} in page~\pageref{fig:moduls}); therefore, it uses only information from the source language. However, during the training of the module, target language information is also used.

Training

To train the lexical selection module, a corpus in the source language and another one in the target language are required; they do not need to be related. Both corpora must be pre-processed before the training. This pre-processing, consisting in analysing the corpora and performing the POS disambiguation, can be done with apertium-prepro\-cess\--cor\-pus\--lex\-tor}.

The training of the module that performs the lexical selection consists of the following tasks:\footnote{The training of the models used for the lexical selection has been automated in all the packages using it. Furthermore, all the software mentioned has its UNIX manual page}

\begin{enumerate}

• Obtain the list of words that will be ignored when performing

lexical selection (\emph{stopwords}). This list can be done manually or using apertium-gen-stopwords-lextor};

• Obtain the list of (source language) words that have more than

one translation in the target language, using apertium-gen-wlist-lextor};

• Translate to the target language all the words obtained in the

previous step, using apertium-gen-wlist-lextor-translation};

• Running apertium-lextor --trainwrd} and using the target

language pre-processed corpus, train a word co-occurrence model for the words obtained in the previous step;

• Running apertium-lextor --trainlch} and using the source

language pre-processed corpus, the dictionaries generated by the programs mentioned in Section~\ref{se:preprocessament} and the word co-occurrence models calculated in the previous step, train a co-occurrence model for each of the translation marks of those words that can have more than one translation in the target language. \end{enumerate}

Use

The word co-occurrence models calculated for each translation mark as described in the previous section provide the information required to perform lexical selection with information from the context.

Lexical selection is done by apertium-lextor --lextor}; the formats used to communicate with the rest of the modules of the translation engine are:




• [Input:] text in the same format as the input for the structural

transfer module, that is, text analysed and disambiguated, with invariable queues of multiwords moved before morphological tags.

• [Output:] text in the same format, but with the translation mark

to be used when executing lexical transfer.

The following example illustrates the input/output formats used by the lexical selector (we have assumed in the example that only the English verb \emph{get} has more than one translation equivalent in the dictionaries):

• Source language text (English): \emph{To get to the city centre}
• Lexical selector input: \verb!^To<pr>$!

\verb!^get<vblex><inf>$! \verb!^to<pr>$! \verb!^the<det><def><sp>$! \verb!^city<n><sg>$! \verb!^centre<n><sg>$!

• Translation marks in the en-ca bilingual dictionary for the verb

\emph{get}: rebre}, agafar}, arribar}, aconseguir D}

• Lexical selector output: \verb!^To<pr>$!

\verb!^get__arribar<vblex><inf>$! \verb!^to<pr>$! \verb!^the<det><def><sp>$! \verb!^city<n><sg>$! \verb!^centre<n><sg>$!