Lextor

When the Apertium system is used to translate between less related languages than the ones dealt with in the first stages of the engine, the question of lexical selection becomes significant, because there are more cases, and more critical, in which a source language word can have more than one different translation in the target language. For this reason we created a new module, the lexical selection module, which deals with this problem.

Before going into its characteristics, we will see how the problems of \emph{multiple equivalence} (the fact of existing more than one possible translation in target language for a source language lexical form) are tackled in Apertium in two ways.

On the one hand, we have the situation where there is no big difference in meaning between the multiple equivalents in the target language, and the fact of choosing one or the other can not lead to any translation error. We could say that between these equivalents there is a synonymy or quasi-synonymy relation. In such a case, the linguist chooses one of the lemmas as a translation (generally the most frequent or usual), and adds a direction restriction to the other lemmas (with the attributes \texttt{LR} or \texttt{RL}) so that they are translated in the opposite direction but not in the direction where there are multiple equivalents.

On the other hand, we have the case where there is a clear difference in meaning between the multiple equivalents, which can lead to translation errors if the inappropriate lemma is chosen. These are the cases dealt with the new lexical selection module. The linguist has to encode entries with the attributes \texttt{slr} or \texttt{srl} described in the next section, thus identifying the different translation options; then, the lexical selection module, by means of statistical methods, chooses the translation which is most suitable in a given context.

Sometimes it is not easy to decide whether a multiple equivalence situation should be solved in one way or the other. For example, if there is difference in the meaning of two or more lemmas in the target language, but we think that the lexical selection module will not be capable of choosing the right translation by means of the context, we will follow the first method: choose a fixed translation (the most general, the most suitable in the maximum number of situations) and add a direction restriction to the rest of translations. In the other cases, we will encode the entries so that the decision is left to the lexical selection module.

When we use an Apertium system without lexical selection module, the only way to add entries with different possible translations is the first one, that is, choosing an only translation and marking the other equivalences with a direction restriction. In the event that we use bilingual dictionaries with multiple translations, encoded with the attributes \texttt{slr} or \texttt{srl}, in a system that does not have any lexical selection module, a style sheet will convert these entries designed for a lexical selection module into entries with direction restrictions \texttt{LR} or \texttt{RL}, so that one of the multiple equivalents (the one chosen as default entry by the linguist) becomes the fixed translation of the source language lemma.

As examples of bilingual equivalencies that should have a direction restriction, we can give the translation pairs \texttt{ca-es} \emph{encara -- aún/todavía} ("still") and \emph{sobtat -- súbito/repentino} ("sudden"), the first one of which could be encoded like this: \begin{alltt} \begin{small}

<e r="LR">

<l>aún</l> <r>encara</r>

</e> <e>

<l>todavía</l> <r>encara</r>

</e> \end{small} \end{alltt}

As examples of the second case (multiple equivalents with big difference in meaning) we have the pairs \texttt{es-ca} \emph{hoja -- full/fulla} ("sheet/leaf") and \emph{muñeca -- nina/canell} ("doll/wrist"), as well as the \texttt{en-ca} examples shown in page \pageref{entrades_lextor}, where it is described how to specify these multiple equivalents in the bilingual dictionary.

\begin{figure} {\footnotesize \setlength{\tabcolsep}{0.5mm} \begin{center} \begin{tabular}{ccccccccc} \\ \parbox{0.95cm}{source language text} \\ $\downarrow$ \\ \framebox{\parbox{1.0cm}{de-for\-matter}} $\rightarrow$ & \framebox{\parbox{0.6cm}{morph. anal.}} $\rightarrow$ & \framebox{\parbox{1.0cm}{POS tagger}} $\rightarrow$ & \framebox{\parbox{0.6cm}{lex. select.}} $\rightarrow$ & \framebox{\parbox{0.85cm}{struct. transf.}} $\rightarrow$ & \framebox{\parbox{0.6cm}{morph. gen.}} $\rightarrow$ & \framebox{\parbox{1.2cm}{post\-generator}} $\rightarrow$ & \framebox{\parbox{1.0cm}{re-for\-matter}} \\ & & & & $\updownarrow$ & & & $\downarrow$ \\ & & & & \framebox{\parbox{0.8cm}{lex. transf.}} & & & \parbox{0.95cm}{target language text} \\ \end{tabular} \end{center} } \caption{The nine modules that build the assembly line in the version

 2 of the machine translation system Apertium.}

\label{fig:moduls} \end{figure}

Figure~\ref{fig:moduls} shows the new assembly line of the version 2 of Apertium.\footnote{This figure substitutes the figure \ref{fg:modules} in page \pageref{pg:modules} which represents the version 1 of Apertium.} \nota{MG: caldria canviar la figura de la pàgina 6 per aquesta d'aquí?} The module in charge of the lexical selection (lexical selector) runs after the part-of-speech tagger and before the structural transfer module; therefore, this new module works only with source language information.

Section~\ref{se:preprocessament} next describes the pre-processing that must be done on a bilingual dictionary containing more than one translation per entry (whether the system uses a lexical selector or not), and Section~\ref{se:lextor} describes how the lexical selector works and how it has to be trained.

\subsection{Pre-processing of the bilingual dictionaries }\label{se:preprocessament}

Bilingual dictionaries have been modified to allow the specification of more than one translation per entry (refer to Section \ref{dic_lextor} to learn how to write such dictionary entries); this fact makes it necessary to pre-process these dictionaries, since the Apertium engine works with compiled dictionaries in which there is only one possible translation for each word.

The pre-processing of dictionaries is done automatically during compilation, therefore the final user does not need to perform any specific action.

\subsubsection{Pre-processing without lexical selection module}

When bilingual dictionaries with multiple equivalents are used in a system where there is no lexical selection module, the pre-processing is done by the application of the style sheet \texttt{translate-to\--de\-fault\--e\-qui\-va\-lent.xsl}. This style sheet turns dictionaries with multiple translations per entry into dictionaries with only one translation per entry; to do this, it chooses as translation the entry marked as default, and adds a direction restriction (\texttt{LR} or \texttt{RL} as applicable) to the other entries, so that they are only translated in the translation direction where there is no equivalent multiplicity. The style sheet is called from the \texttt{Makefile}.

To put an example, the result of applying the style sheet on the first three entries shown in page \pageref{entrades_lextor} is the following:

\begin{alltt} \begin{small} <e>

<l>flat</l> <r>pis</r>

</e>

<e r="LR">

<l>floor</l> <r>pis</r>

</e>

<e r="RL">

<l>floor</l> <r>terra</r>

</e> \end{small} \end{alltt}

\subsubsection{Preprocessing with lexical selection module}

If the Apertium system works with a lexical selection module, the bilingual dictionary must be pre-processed in order to obtain: \begin{itemize} \item a monolingual dictionary that, for each source language word (for example \emph{look}) delivers all the possible translation marks or equivalents (\texttt{look\_\_mirar D} and \texttt{look\_\_semblar}); this dictionary will be used by the lexical selection module; and

\item a new bilingual dictionary that, given a word with the lexical selection already done (for example \texttt{look\_\_semblar}) delivers the translation (\emph{semblar}); this will be the bilingual dictionary to be used in the lexical transfer.

\end{itemize}

This pre-processing is automatically done by means of the following software during dictionary compilation: \begin{itemize} \item \texttt{apertium-gen-lextormono}, that receives three parameters:

 \begin{itemize}
 \item the translation direction for which you want to generate the
 monolingual dictionary used in the lexical selection; \texttt{lr}
 for the translation left to right, and \texttt{rl} for the
 translation right to left;
 \item the monolingual dictionary to be pre-processed; and
 \item the file where the output monolingual dictionary has to be
 written.
 \end{itemize}

\item \texttt{apertium-gen-lextorbil}, that receives three parameters:

 \begin{itemize}
 \item the translation direction (\texttt{lr} or \texttt{rl}) for
   which you want to generate the bilingual dictionary to be used by
   the lexical transfer module;
 \item the bilingual dictionary to be pre-processed; and
 \item the file where the output bilingual dictionary has to be
 written.
 \end{itemize}

\end{itemize}

\subsection{Execution of the lexical selection module}\label{se:lextor}

The module responsible for the lexical selection runs after the part-of-speech tagger and before the structural transfer (see Figure~\ref{fig:moduls} in page~\pageref{fig:moduls}); therefore, it uses only information from the source language. However, during the training of the module, target language information is also used.

\subsubsection{Training}\label{se:entrenament}

To train the lexical selection module, a corpus in the source language and another one in the target language are required; they do not need to be related. Both corpora must be pre-processed before the training. This pre-processing, consisting in analysing the corpora and performing the POS disambiguation, can be done with \texttt{apertium-prepro\-cess\--cor\-pus\--lex\-tor}.

The training of the module that performs the lexical selection consists of the following tasks:\footnote{The training of the models used for the lexical selection has been automated in all the packages using it. Furthermore, all the software mentioned has its UNIX manual page}

\begin{enumerate} \item Obtain the list of words that will be ignored when performing lexical selection (\emph{stopwords}). This list can be done manually or using \texttt{apertium-gen-stopwords-lextor}; \item Obtain the list of (source language) words that have more than one translation in the target language, using \texttt{apertium-gen-wlist-lextor}; \item Translate to the target language all the words obtained in the previous step, using \texttt{apertium-gen-wlist-lextor-translation}; \item Running \texttt{apertium-lextor --trainwrd} and using the target language pre-processed corpus, train a word co-occurrence model for the words obtained in the previous step; \item Running \texttt{apertium-lextor --trainlch} and using the source language pre-processed corpus, the dictionaries generated by the programs mentioned in Section~\ref{se:preprocessament} and the word co-occurrence models calculated in the previous step, train a co-occurrence model for each of the translation marks of those words that can have more than one translation in the target language. \end{enumerate}

\subsubsection{Use}\label{se:us}

The word co-occurrence models calculated for each translation mark as described in the previous section provide the information required to perform lexical selection with information from the context.

Lexical selection is done by \texttt{apertium-lextor --lextor}; the formats used to communicate with the rest of the modules of the translation engine are:

\begin{description} \item [Input:] text in the same format as the input for the structural transfer module, that is, text analysed and disambiguated, with invariable queues of multiwords moved before morphological tags. \item [Output:] text in the same format, but with the translation mark to be used when executing lexical transfer. \end{description}

The following example illustrates the input/output formats used by the lexical selector (we have assumed in the example that only the English verb \emph{get} has more than one translation equivalent in the dictionaries): \begin{itemize} \item Source language text (English): \emph{To get to the city centre} \item Lexical selector input: \verb!^To<pr>$! \verb!^get<vblex><inf>$! \verb!^to<pr>$! \verb!^the<det><def><sp>$! \verb!^city<n><sg>$! \verb!^centre<n><sg>$! \item Translation marks in the en-ca bilingual dictionary for the verb \emph{get}: \texttt{rebre}, \texttt{agafar}, \texttt{arribar}, \texttt{aconseguir D} \item Lexical selector output: \verb!^To<pr>$! \verb!^get__arribar<vblex><inf>$! \verb!^to<pr>$! \verb!^the<det><def><sp>$! \verb!^city<n><sg>$! \verb!^centre<n><sg>$! \end{itemize}