View source for Курсы машинного перевода для языков России/Session 6

{{TOCD}}

While the basic structural transfer described in [[/Session 5|session 5]] is enough to deal with the majority of frequent divergences between closely related languages (for example Bashkir and Tatar or Finnish and Kven), when working with languages with more divergent morphology and syntax, a more powerful structural transfer module is necessary.  This session describes the Apertium 3+ level transfer system which was designed to allow easier treatment of longer patterns, and more divergent languages.

==Theory==

===Chunking-based transfer===
The typical implementation of the chunking-based transfer consists of three modules: a chunker, an interchunk and a postchunk. This design can be extended to contain two or more interchunk modules if needed.
====Chunker====

The idea of the ''chunker'' is to extend the existing transfer rules to allow sequences of lexical units to be grouped. These groups are called ''chunks'' . A typical chunk might be for grouping nominals, doing concordance, inserting or deleting words, and performing local reordering, for example,

{|class=wikitable
|-
! Input pattern                   !! Example    !! Output chunk                             !! Example
|-
| <code>nom</code>                || ҫурт    || <code>SN{nom}</code>              || дом 
|-
| <code>adj nom</code>            || Mooi ҫурт    || <code>SN{nom adj}</code>          || красивый дом
|-
| <code>nom</code>                || ҫуртам    || <code>SN{det nom}</code>          || мой дом
|-
| <code>num nom</code>            || Twee ҫурт    || <code>SN{num nom}</code>          || два дома
|-
| <code>num nom</code>            || Twee ҫурт    || <code>SN{num nom}</code>          || пять домов
|-
| <code>det adj nom</code>        || ҫуртсем    || <code>SN{det nom adj}</code>      || красивые домы
|-
| <code>det adv adj nom</code>    || De erg onbeleefde ҫуртсем    || <code>SN{det nom adv adj}</code>  || пять очень красивых домов 
|-
| <code>det num adj nom</code>    || De twee onbeleefde ҫуртсем    || <code>SN{det num nom adj}</code>  || dos дома maleducados
|-
|}

Where <code>nom</code> = noun, <code>adj</code> = adjective, <code>num</code> = numeral, <code>det</code> = determiner, <code>SN</code> = noun phrase.

The same process also works for verb chunks:

{|class=wikitable
|-
! Input pattern                   !! Example          !! Output chunk                             !! Example
|- 
| <code>verb</code>               || komt             || <code>V{verb}</code>                     || Viene
|-
| <code>verb neg_adv</code>       || komt niet        || <code>V{neg_adv verb}</code>             || No viene
|-
| <code>zijn neg_adv pp</code>    || is niet gekomen  || <code>V{neg_adv haber pp}</code>         || No ha venido
|-
| <code>aux neg_adv inf</code>    || zal niet komen   || <code>V{neg_adv verb}</code>             || No vendrá
|}


Thus, if we want to concord a noun phrase with a main verb, we can just use one rule (for <code>SN V</code>) in the second module of the transfer (the ''interchunk'') instead of having separate rules for <code>nom verb</code>, <code>adj nom verb</code>, <code>det adj nom verb</code>, etc. 

An important thing to remember is that chunks cannot be nested (i.e. a chunk may not contain another chunk). In some circumstances, and with some effort they can be merged at the interchunk stage &mdash; for example to join together one or more coordinated noun phrases, but not nested.

It should be noted that lexical forms are translated into the target language in this first module; the subsequent modules only work with lexical forms in the target language.

====Interchunk====

Once these chunks are made, the next module ''interchunk'' allows operations to be made between chunks as if they were lexical units in themselves: chunks are used as a level of abstraction, so that equivalent words and phrases can be translated using the same rules.

Consider the Spanish sentences: 

* ''El hombre vio el perro'', 
* ''El hombre ha visto el perro'', 
* ''El hombre alto podría haber visto el perro blanco''

Each of these sentences would produce the same output chunks in the chunker: 'nominal chunk' 'verbal chunk' 'nominal chunk' &mdash; which interchunk then performs a second set of structural transformations on.

As well as gender concordance and word reordering, this allows gender 'detection'. Consider the Spanish word 'doctor', which has a feminine form 'doctora'. In the Spanish analyser, 'doctora' is analysed as a form of 'doctor', rather than as a separate word in its own right, and in the bilingual dictionary it has the tag 'GD' attached. In the Dutch sentence 'Maria is een dokter', the chunker would give the following output:

<pre>
^Nom<SN><UNDET><f><sg>{^Maria<np><ant><3><4>$}$ 
^zijn<Vcop><vbser><pri><p3><sg>{^ser<vbser><3><4><5>$}$ 
^det_nom<SN><DET><GD><sg>{^uno<det><ind><3><4>$ ^doctor<n><3><4>$}$
</pre>

The format of chunks is much like that of lexical units, <code>^</code> indicates the start, and <code>$</code> the end. The difference being that a chunk may contain other lexical units within <code>{</code> and <code>}</code>.

The lexical units inside the chunk (between the <code>{</code> and <code>}</code> signs) cannot be accessed or modified in the interchunk; here you can only access or modify elements from the description of the chunk, which is the part after <code>^</code> and before the first <code>{</code>. The description of the chunk contains the lemma of the chunk (like <code>det_nom</code> in the previous example) and the morphological tags of the chunk (which for <code>det_nom</code> are {{tag|SN><DET><GD><sg}}). These tags can be linked with the lexical forms inside the chunk: this is the reason for the numbers {{tag|3}} and {{tag|4}} in the lexical forms of the <code>det_nom</code> chunk: they are linked with the third and fourth tags of the chunk ({{tag|GD}} and {{tag|sg}}) and will be substituted for them in the postchunk module.

Interchunk has a rule for 'nominal chunk' 'copula' 'nominal chunk', which copies the gender from the first nominal chunk to the second, replacing the 'GD' tag; in this example, giving it the feminine value:

<pre>
^Nom<SN><PDET><f><sg>{^Maria<np><ant><3><4>$}$ 
^zijn<Vcop><vbser><pri><p3><sg>{^ser<vbser><3><4><5>$}$ 
^det_nom<SN><DET><f><sg>{^uno<det><ind><3><4>$ ^doctor<n><3><4>$}$
</pre>

The postchunk module will assign this tag to the determiner and the noun inside the chunk.

====Postchunk====

Postchunk allows us to take the output of interchunk, and once again operate on the contents. 

Changes made on the chunks in the interchunk module, will be applied to the contents of the chunk: tags containing a number will be substituted for the value of the corresponding tag outside of the chunk. The ''postchunk'' module removes the chunk ''lemma'' and tags, and leaves the output as a sequence of lexical units.

In the ''Maria is een dokter'' example, {{tag|UNDET}} changed to {{tag|PDET}}. This is an indicator to the postchunk module that this ''may'' be a chunk which takes a definite article in Spanish (in this particular case, it's not).

Postchunk operates on a single chunk at a time. In addition to the <tt>clip</tt> elements which refer to individual words contained in the chunk, there is also a <tt>clip</tt> numbered 0 (zero), which allows us to access information from the chunk lemma, which can be used to take information from "outside" the chunk (changed in interchunk) to the words inside. Also, because the number of words in a chunk may vary, there is an element, <tt>lu-count</tt>, which allows us to test how many words the chunk contains, and act accordingly.

==Practice==

For the practice section, we are going to look at how a transfer is performed in three stages by the Apertium Spanish&mdash;Italian pair, <code>apertium-es-it</code>, and then describe a transfer rule in terms of three or more levels. So change directory to <code>apertium-es-it</code> and make sure the pair is compiled.

===Looking at three-stage transfer===

We're going to translate the sentence ''Los zapatos nuevos son demasiado pequeños.'' from Spanish to Italian and follow the translation process through the three levels.

====Input====

First we morphologically analyse and tag the text:

<pre>
$ echo "Los zapatos nuevos son demasiado pequeños" | apertium -d . es-it-tagger
^El<det><def><m><pl>$ ^zapato<n><m><pl>$ ^nuevo<adj><m><pl>$ ^ser<vblex><pri><p3><pl>$
^demasiado<adv>$ ^pequeño<adj><m><pl>$^.<sent>$
</pre>

====Chunker====

Then output of the part-of-speech tagger is passed to the first level of transfer:

<pre>
$ echo "Los zapatos nuevos son demasiado pequeños" | apertium -d . es-it-chunker
^Det_nom_adj<SN><f><pl>{^il<det><def><2><3>$ ^scarpa<n><2><3>$ ^nuovo<adj><2><3>$}$ 
^verb<SV><vbser><pri><p3><pl>{^essere<vbser><pri><p3><5>$}$ 
  ^adv_adj<SA><m><pl>{^troppo<adv>$ ^piccolo<adj><2><3>$}$^punt<sent>{^.<sent>$}$
</pre>

There are three rules applied by the first-level transfer module:

* <code>REGLA: DET ADJ NOM</code>: This rule matches a determiner, followed by an adjective followed by a noun. It creates a new nominal chunk and sets the gender and number of the chunk to be those of the head noun. The tags inside the chunk for gender and number are replaced by pointers to the tags in the chunk.
* <code>REGLA: VERB</code>: This is the default verb rule, it matches any verb, and performs some local changes. For example, changing the future subjunctive to the imperfect subjunctive (in this case not applicable). It outputs the verb, verb type and other information in the chunk, and links the number to the chunk for possible future concordance operations.
* <code>REGLA: ADV ADJ</code>: This rule matches an adverb followed by an adjective. It outputs an adjective chunk and links the gender and number of the adjective to the chunk.

Note that after the first stage of transfer there is an agreement error between the subject and the predicate. The gender of the subject ''Los zapatos nuevos'' has been changed from masculine to feminine, but that of the predicate ''demasiado pequeños'' has not. 

====Interchunk====

<pre>
$ echo "Los zapatos nuevos son demasiado pequeños" | apertium -d . es-it-interchunk
^Det_nom_adj<SN><f><pl>{^il<det><def><2><3>$ ^scarpa<n><2><3>$ ^nuovo<adj><2><3>$}$ 
^verb<SV><vbser><pri><p3><pl>{^essere<vbser><pri><p3><5>$}$ 
  ^adv_adj<SA><f><pl>{^troppo<adv>$ ^piccolo<adj><2><3>$}$^punt<sent>{^.<sent>$}$
</pre>

One rule is applied in the interchunk module:

* <code>REGLA: SN ser SA</code>: The rule matches a nominal chunk (<code>SN</code>) followed by the verb ''essere'' and an adjective chunk (<code>SA</code>). It contains a macro <code>concorda_SN_SA</code> which checks to see if the adjective chunk agrees in gender and number with the nominal chunk. 

We can see that in the output of interchunk, the adjective gender has been changed to that of the nominal chunk.

====Postchunk====

The final module of transfer takes the chunks output by the interchunk module, and replaces the linked tags ({{tag|2}}, {{tag|3}}, etc.) with their values from the chunk.

<pre>
$ echo "Los zapatos nuevos son demasiado pequeños" | apertium -d . es-it-postchunk
^Il<det><def><f><pl>$ ^scarpa<n><f><pl>$ ^nuovo<adj><f><pl>$ ^essere<vbser><pri><p3><pl>$ 
^troppo<adv>$ ^piccolo<adj><f><pl>$^.<sent>$
</pre>

Now the sentence is ready to be morphologically generated.

====Output====

<pre>
$ echo "Los zapatos nuevos son demasiado pequeños" | apertium -d . es-it
Le scarpe nuove sono troppo piccole
</pre>

===Describing a multi-stage transfer rule===

The important thing to work out when writing a multi-stage transfer rule is how to split the rule between the different parts of transfer. For example, local reorderings (at the level of 1&mdash;5 words) should probably be done in the first stage. The chunks should be in some way thematic, so for example, finite verbs should probably not be chunked with adjectives or nominals.

==Further reading==

* Ginestí i Rosell, M. (ed.) (2007) [https://wiki.apertium.org/w/images/d/d0/Apertium2-documentation.pdf  Documentation of the Open-Source Shallow-Transfer Machine Translation Platform ''Apertium'']

{{Sessions}}

[[Category:Session 6|*]]