Курсы машинного перевода для языков России/Раздел 6

Система базового структурного переноса, описанного в session 5 достаточно для работы с большинством расхождений, которые имеются в родственных языках (например, башкирский и татарский, финский и квен), при работе с языками, которые имеют больше морфологических и синтаксических расхождений, необходимо использовать более мощную систему структурного переноса. В данном разделе мы познакомим вас с системой уровня Apertium 3+, которая была разработана для работы с более длинными шаблонами и языками, которые сильнее отличаются друг от друга.

Теория

Передача путем разбивки на блоки Chunking-based transfer

Традиционное применение данного метода состоит из трех модулей: блок a chunker, между блоками и послеan interchunk and a postchunk. При необходимости данная модель может быть расширена до двух и более модулей intertchunk.

Chunker

Идея chunker - расширить существующие правила переноса чтобы было возможно сгруппировать лексические единицы. Эти группы называются блоки . Блок может объединять имена, делать согласование, вставлять и удалять слова, и выполнять перестановку слов, например,

Input pattern	Example	Output chunk	Example
nom	ҫурт	SN{nom}	дом
adj nom	хитре ҫурт	SN{nom adj}	красивый дом
nom	ҫуртӑм	SN{det nom}	мой дом
det nom	манăн ҫурт	SN{det nom}	мой дом
det nom	манăн ҫуртӑм	SN{det nom}	мой дом
num nom	икĕ ҫурт	SN{num nom}	два дома
num nom	пилĕ ҫурт	SN{num nom}	пять домов
adj nom	хитре ҫуртсем	SN{adj nom}	красивые домы
adv adj nom	питĕ хитре ҫурт	SN{adv adj nom}	очень красивый дом
num adv adj nom	пилĕ питĕ хитре ҫурт	SN{num adv adj nom}	пять очень красивых домов

Where nom = noun, adj = adjective, num = numeral, det = determiner, SN = noun phrase.

The same process also works for verb chunks:

Input pattern	Example	Output chunk	Example
verb	вулать	V{verb}	читает
verb	вуламасть	V{neg_adv verb}	не читает
verb	вуларĕ	V{verb}	читал
verb	вуламĕ	V{aux inf}	будет читать
verb	вуламарĕ	V{neg_adv aux inf}	не будет читать
verb	вуласшăн	V{aux part inf}	хотел бы говорить
verb	вулӑттӑм	V{verb part}	говорил бы
adv verb	ан вула !	V{adv verb}	не читай !
ger verb	вулама пуçлать	V{verb inf}	начинает читать.

Thus, if we want to concord a noun phrase with a main verb, we can just use one rule (for SN V) in the second module of the transfer (the interchunk) instead of having separate rules for nom verb, adj nom verb, det adj nom verb, etc.

An important thing to remember is that chunks cannot be nested (i.e. a chunk may not contain another chunk). In some circumstances, and with some effort they can be merged at the interchunk stage — for example to join together one or more coordinated noun phrases, but not nested.

It should be noted that lexical forms are translated into the target language in this first module; the subsequent modules only work with lexical forms in the target language.

Interchunk

Once these chunks are made, the next module interchunk allows operations to be made between chunks as if they were lexical units in themselves: chunks are used as a level of abstraction, so that equivalent words and phrases can be translated using the same rules.

As well as gender concordance and word reordering, this allows person 'detection' — for example to concord a verb in the past tense in Chuvash with the pronoun in the sentence. In the Russian sentence Я вчера читалa, the chunker would give the following output:

^pron<SN><p1><mf><sg><nom>{^Эпĕ<prn><pers><2><3><4><5>$}$ 
^adv<ADV>{^ĕнер<adv>$}$
^verb<SV><imperf><tv><evid><PD><f><sg>{^вула<v><3><4><5><7>$}$ 

The format of chunks is much like that of lexical units, ^ indicates the start, and $ the end. The difference being that a chunk may contain other lexical units within { and }.

The lexical units inside the chunk (between the { and } signs) cannot be accessed or modified in the interchunk; here you can only access or modify elements from the description of the chunk, which is the part after ^ and before the first {. The description of the chunk contains the lemma of the chunk (like pron in the previous example) and the morphological tags of the chunk (which for pron are <SN><p1><mf><sg><nom>).

These tags can be linked with the lexical forms inside the chunk: this is the reason for the numbers <5> and <7> in the lexical forms of the verb chunk: they are linked with the fifth and seventh tags of the chunk (<PD> and <sg>) and will be substituted for them in the postchunk module.

Interchunk has a rule for 'nominal chunk' 'adv' 'verb chunk', which copies the person from the first nominal chunk to the verb chunk, replacing the 'PD' tag; in this example, giving it the <p1> (first person) value:

^pron<SN><p1><mf><sg><nom>{^Эпĕ<prn><pers><2><3><4><5>$}$ 
^adv<ADV>{^ĕнер<adv>$}$
^verb<SV><imperf><tv><evid><p1><f><sg>{^вула<v><3><4><5><7>$}$ 

The postchunk module will assign this tag to the verb inside the chunk.

Postchunk

Postchunk allows us to take the output of interchunk, and once again operate on the contents.

Changes made on the chunks in the interchunk module, will be applied to the contents of the chunk: tags containing a number will be substituted for the value of the corresponding tag outside of the chunk. The postchunk module removes the chunk lemma and tags, and leaves the output as a sequence of lexical units.

Postchunk operates on a single chunk at a time. In addition to the clip elements which refer to individual words contained in the chunk, there is also a clip numbered 0 (zero), which allows us to access information from the chunk lemma, which can be used to take information from "outside" the chunk (changed in interchunk) to the words inside. Also, because the number of words in a chunk may vary, there is an element, lu-count, which allows us to test how many words the chunk contains, and act accordingly.

Practice

For the practice section, we are going to look at how a transfer is performed in three stages by the Apertium Tatar—Kyrgyz pair, apertium-tt-ky, and then describe a transfer rule in terms of three or more levels. So change directory to apertium-tt-ky and make sure the pair is compiled.

Looking at three-stage transfer

We're going to translate the sentence Әхмәт тиз генә иске зур бер агачка йөгерә. from Tatar to Kyrgyz and follow the translation process through the three levels.

Input

Because we don't yet have a full translator for Tatar and Kyrgyz, we're going to use some preprepared input from the Tatar and Bashkir pair.

$ cat input 
^Әхмәт<np><ant><m><nom>$ ^тиз<adv>$ ^гына<postadv>$ ^иске<adj>$ ^зур<adj>$ ^бер<det><ind>$ 
^агач<n><dat>$ ^йөгер<v><iv><pres><p3><sg>$^..<sent>$

Chunker

Then output of the part-of-speech tagger is passed to the lexical transfer, and then the first level of transfer:

$ cat input | lt-proc -b tt-ky.autobil.bin | apertium-transfer -b apertium-tt-ky.tt-ky.t1x tt-ky.t1x.bin 

^ant<SN>{^Акмат<np><ant><m><nom>$}$ ^adv<ADV>{^катуу<adv>$ ^гана<postadv>$}$ 
^a_a_d_n<SN><dat>{^эски<adj><pst>$ ^чоң<adj><pst>$ ^бир<det><ind>$ ^дарак<n><2>$}$ 
^чурка<V>{^чурка<v><iv><prt_perf>$ ^бар<v><iv><prt_impf>$ ^жат<vaux><aor><p3><sg>$}$^sent<SENT>{^..<sent>$}$

There are four rules applied by the first-level transfer module:

• ПРАВИЛО: NP-ANT: This rule matches an anthroponym (person's first name). It creates a new nominal <SN> chunk.
• ПРАВИЛО: ADV POSTADV: This rule matches a sequence of adverb and postadverb, it outputs an adverbial chunk <ADV> containing the two lexical units.
• ПРАВИЛО: ADJ ADJ DET NOM: This rule matches a sequence of two adjetives, a determiner and a noun. These are put inside a nominal chunk <SN> and the case of the chunk is set to the case of the noun. A pointer <2> is put on the noun so that when the case of the chunk is changed, it will be propagated inside.
• ПРАВИЛО: V-PRES: This is the default present tense verb rule, it matches any verb in the present tense. It currently changes the synthetic present in Tatar into a progressive present tense with an auxiliary verb in Kyrgyz. WHY???

Note that after the first stage of transfer there are a couple of problems. The tense is correct, but the case of the noun is wrong, and the adverbial is in the wrong place. In Kyrgyz it should come before the verb.

Interchunk

$ cat input | lt-proc -b tt-ky.autobil.bin | apertium-transfer -b apertium-tt-ky.tt-ky.t1x tt-ky.t1x.bin |\
   apertium-interchunk apertium-tt-ky.tt-ky.t2x tt-ky.t2x.bin 

^ant<SN>{^Акмат<np><ant><m><nom>$}$ ^a_a_d_n<SN><acc>{^эски<adj><pst>$ ^чоң<adj><pst>$ ^бир<det><ind>$ ^дарак<n><2>$}$ 
^post<POST>{^көздөй<post>$}$ ^adv<ADV>{^катуу<adv>$ ^гана<postadv>$}$ 
^чурка<V>{^чурка<v><iv><prt_perf>$ ^бар<v><iv><prt_impf>$ ^жат<vaux><aor><p3><sg>$}$^sent<SENT>{^..<sent>$}$

One rule is applied in the interchunk module:

• ПРАВИЛО: ADV SN V: The rule matches an adverbial chunk (ADV) followed by a nominal chunk (SN) and then a verbal chunk (V). It contains a macro conv_arg1 which adjusts the case of the nominal chunk, and outputs a postposition depending on the lemma of the verbal chunk. It also switches the position of the nominal chunk and the adverbial chunk, placing the adverbial before the verb.

We can see that in the output of interchunk, the adverbial has been moved and the nominal chunk is in the correct case with a postposition.

Postchunk

The final module of transfer takes the chunks output by the interchunk module, and replaces the linked tag (e.g. <2>) with its value from the chunk.

$ cat input | lt-proc -b tt-ky.autobil.bin | apertium-transfer -b apertium-tt-ky.tt-ky.t1x tt-ky.t1x.bin |\
  apertium-interchunk apertium-tt-ky.tt-ky.t2x tt-ky.t2x.bin | apertium-postchunk apertium-tt-ky.tt-ky.t3x tt-ky.t3x.bin 

^Акмат<np><ant><m><nom>$ ^эски<adj><pst>$ ^чоң<adj><pst>$ ^бир<det><ind>$ ^дарак<n><acc>$ ^көздөй<post>$ ^катуу<adv>$ 
^гана<postadv>$ ^чурка<v><iv><prt_perf>$ ^бар<v><iv><prt_impf>$ ^жат<vaux><aor><p3><sg>$^..<sent>$

Now the sentence is ready to be morphologically generated. The file tr-ky.autogen.hfst can be copied from the apertium-tr-ky pair in trunk/.

Output

$ cat input | lt-proc -b tt-ky.autobil.bin | apertium-transfer -b apertium-tt-ky.tt-ky.t1x tt-ky.t1x.bin |\
   apertium-interchunk apertium-tt-ky.tt-ky.t2x tt-ky.t2x.bin | apertium-postchunk apertium-tt-ky.tt-ky.t3x tt-ky.t3x.bin |\
   hfst-proc -g tr-ky.autogen.hfst 

Акмат эски чоң бир даракты көздөй катуу гана чуркап бара жатат.

Describing a multi-stage transfer rule

The important thing to work out when writing a multi-stage transfer rule is how to split the rule between the different parts of transfer. For example, local reorderings (at the level of 1—5 words) should probably be done in the first stage. The chunks should be in some way thematic, so for example, finite verbs should probably not be chunked with adjectives or nominals.

Further reading

• Ginestí i Rosell, M. (ed.) (2007) Documentation of the Open-Source Shallow-Transfer Machine Translation Platform Apertium