Курсы машинного перевода для языков России/Session 5

The aim of this session is to give an introduction to the idea of contrastive morphological and syntactic analysis, and show how basic (that is, single level) morphological and syntactic transfer rules can be made in Apertium.

Theory

Contrastive analysis

Contrastive analysis is the process of examining two or more languages together to find out what kind of features they share, and how they are distinguished. When working on shallow-transfer machine translation, we can consider for example, both morphological contrasts and syntactic contrasts.

Morphological contrasts

Morphological contrasts are differences in which morphological features are expressed in each language, and how they differ. For example, if one language expresses definiteness morphologically, and another doesn't, or one language has case or gender while the other doesn't. Or if the case/gender inventories differ between the languages. Examples:

• Kazakh has no grammatical gender where Russian has three (masculine, feminine and neuter). Gender in Russian plays a rôle in agreement within noun phrases.
  • Жаңа әуежай^(kk) → Новый аэропорт^(ru)
• Russian has two numbers (singular and plural), where Mansi has three (singular, dual and plural).
  • Я читаю эту книгу^(ru) → Ам ти книга ловиньтилум^(mns)
  • Я читаю эти две книги^(ru) → Ам ти книгаг ловиньтиягум^(mns)
  • Я читаю эти книги^(ru) → Ам ти книгат ловиньтиянум^(mns)
• Hungarian has a large number of cases which don't exist in Finnish (for example the temporal case -kor, which can be translated into Finnish by the ablative, adessive, or essive). Case in Hungarian and Finnish does not play a rôle in noun-phrase agreement.
  • Ötkor^(hu) → Viideltä^(fi)
  • Éjfélkor^(hu) → Keskiyöllä^(fi)
  • Karácsonykor^(hu) → Jouluna^(fi)

Syntactic contrasts

Syntactic contrasts are differences in how the syntax of the languages works, for example the existence or not of articles, ordering of clitic pronouns, case inventories, analytic verb tenses, usage of subject pronouns, etc.

• Abkhaz has a postfixed indefinite article, and a prefixed definite article where Russian does not.
  • Сара кьылак акәац аасхәоит^(ab) → Я покупаю килограмм мяса^(ru)
  • Иара амшын ахь дцоит.^(ab) → Он идёт в море.^(ru)
• Russian has compulsory subject pronouns, where Hungarian does not,
  • Я сплю.^(ru) → Alszom.^(hu)
• Chuvash has an ablative case, where Russian uses a preposition and a different case.
  • Я получил письмо от друга.^(ru) → Юлташран ҫыру илтӗм.^(cv)

Transfer

Transfer is the process of altering the intermediate representation of a source language into that of the target language. In Apertium, transfer works on an intermediate representation of lemmas and tags specifying morphological features (lexical forms).

Basic operations

When thinking about implementing transfer rules, it is often worth trying to think in terms of some very basic operations.

A) Insertion

Insertion is the operation of adding a new tag or word, for example to translate a noun from Dungan to Russian, a tag denoting case (nominative, accusative, ...) would need to be inserted, as Dungan does not have this morphological feature. Many times translation also requires the insertion of a word. For example when translating the imperative from Turkish to Chuvash, the adverb ан needs to be added to the beginning: Yeme! → Ан çи!.

B) Deletion

Deletion is the opposite operation to insertion, a word or tag needs to be removed, for example as in the previous example, once the adverb has been inserted, the negative would need to be deleted (or not output) from the following verb, as it is the adverb which provides the negation.

C) Substitution

Substitution is replacing one tag with another tag, such as for example changing the gender of a word. This is often done in the transfer lexicon, as we saw in the last session, but it can also take place in transfer. For example if you are translating from Russian to Bashkir, мои брюки, the word брюки would be marked with a special tag ND (number to be determined) in the transfer lexicon, and then the transfer rules would substitute this with the appropriate number.

D) Reordering

Reordering is changing the order of tags or words. For example the order of number and possessive when translating from Turkish into Chuvash kitap·lar·ım → кӗнеке·м·сем.

E) Combining operations

A lot of the time, a combination of these operations is required to make a transfer rule, for example when translating from language with a case for a given meaning into a language with a preposition and a case for the same meaning, two operations are required, first the word for preposition needs to be inserted before the noun, and secondly the feature denoting case on the noun needs to be changed.

Transfer structure

Files containing structural transfer rules in Apertium are laid out in the following fashion:

<transfer default="chunk">
  <section-def-cats>
    <def-cat n="adj_or_pp">
      <cat-item tags="adj.*">
      <cat-item tags="vblex.pp.*">
    </def-cat>
    ...
  </section-def-cats>
  <section-def-attrs>
    <def-attr n="nbr">
      <attr-item n="sg"/>
      <attr-item n="pl"/>
    </def-attr>
    ...
  </section-def-attrs>
  <section-def-vars>
    <def-var n="number"/> 
    ...
  </section-def-vars>
  <section-rules>
    ...
  </section-rules>
</transfer>

This is the minimal layout. The four sections are:

• <section-def-cats>: Contains one or more <def-cat> entries. These specify patterns that can be matched by the transfer rules, and may match either tag sequences, or lemmas. The <def-cat> entry above for adj_or_pp matches any lexical unit in which the tags start with <adj> or <vblex><pp>. This might be useful where two categories behave similarly (for example adjectives and past particiles in Spanish).
• <section-def-attrs>: Contains one or more <def-attr> entries. These list possible tags corresponding to a feature. For example, in this case the feature of nbr "number" may be one of two tags <sg> "singular" or <pl> "plural". When we use the <clip> tag, to extract a part, this allows us to define our own parts, in addition to the default parts.
• <section-def-vars>: Contains variable definitions. Variables are used to pass information between rules. For example we might want to keep track of the last gender or number that we have seen, or the lemma of the last finite verb.
• <section-rules>: Contains the rules.

Default parts

In addition to the parts that are defined in section-def-attrs, the transfer module predefines a number of default parts of each lexical unit.

To more easily demonstrate the default parts that are available, we'll use the follow example:

^правило# по безопасности<n><nt><nn><pl>$

• whole: as the name suggests, this provides the whole content of the lexical unit.
• lem: the lemma: in this case, правило# по безопасности
• tags: all of the tags, in one unit: <n><nt><nn><pl>
• lemh: the lemma head; the part of the multiword which inflects: правило
• lemq: the lemma queue; the rest of a multiword (the <g></g> part in the dictionaries): # по безопасности

Practice

For this practical, we're going to use the Turkish--Chuvash language pair in Apertium, you could also choose another pair with single-level transfer (such as Tatar--Bashkir). So, go to the directory apertium-cv-tr and open the file of transfer rules for Turkish to Chuvash apertium-cv-tr.tr-cv.t1x, in other pairs this might be a .t1x file.

Describe a transfer rule

The objective of this exercise is to describe the behaviour of an existing transfer rule given the input sentence in Turkish: Yeme!. First we're going to give some preliminaries for how transfer rules are structured, then we're going to look at an actual rule and describe it.

Transfer rules in Apertium are made up of two principle parts. A pattern — that is a sequence of categories which will be matched, and an action, which contains operations that are carried out on the patterns matched. An overall schema for a rule might look like the following:

<rule>
 <pattern>
        ...
 </pattern>
 <action>
        ...
 </action>
</rule>

An overview of the meanings of the tags which are used in the example below are given here, along with references to the page of the documentation where a full description can be found.

Name	Doc. ref.	Description
<rule>	§3.5.4.18	Starts a new rule, it contains at the highest level a pattern and an action.
<pattern>	§3.5.4.19	Contains one or more pattern-item tags that define the pattern of lexical forms to be matched.
<pattern-item>	§3.5.4.20	Contains references to patterns of lexical forms defined in the section-def-cats part of the rule file.
<action>	§3.5.4.21	Contains tags defining the actions that should take place when a sequence of lexical units matching pattern is matched.
<choose>	§3.5.4.24	Contains one or more when statements which define different actions depending on different conditions, and optionally an otherwise statement, which defines a default action if none of the whens are matched.
<when>	§3.5.4.25	Specifies a condition and an action to take if that condition is fulfilled. The condition is in a test tag, and the action comes after.
<otherwise>	§3.5.4.26	A default condition, which contains actions that are run if none of the when conditions in a choose block are matched.
<test>	§3.5.4.27	Describes a condition. For example "test if the number of the first pattern matched is singular".
<or>	§3.5.4.28	A tag allowing more than one condition to be matched. This tag is a Boolean operator, other Boolean operators available are: <and> and <not>
<equal>	§3.5.4.28	Tests if two strings or tags are equal.
<clip>	§3.5.4.29	Extracts the part of a lexical unit corresponding to an attribute as defined in the section-def-attrs section of the transfer file. The part to be extracted is specified in the part attribute.
<lit-tag>	§3.5.4.31	Generates a string enclosed in < >, that is to say, a tag.
<lit>	§3.5.4.30	Generates a string.
<out>	§3.5.4.40	Contains everything that will be output by the rule.
<lu>	§3.5.4.41	Encloses the contents in ^ and $, that is defines the contents as a lexical unit.
<b>	§3.5.4.46	Outputs a space, or the formatting contained at the given position.
<get-case-from>	§3.5.4.34	Converts the case (e.g. upper, lower) of whatever is enclosed in it to the case of the word marked by the pos attribute.

The rule we're going to describe is regla: v_neg_imp, so search for it in the file. Here is what the rule looks like:

    <rule comment="regla: v_neg_imp (yeme! → ан ҫи!)">
      <pattern>
        <pattern-item n="v_neg_imp"/>
      </pattern>
      <action>
        <let>
          <clip pos="1" side="tl" part="a_neg"/>
          <lit v=""/>
        </let>
        <out>
          <lu>
            <lit v="ан"/>
            <lit-tag v="adv"/>
          </lu>
          <b/>
          <lu>
            <clip pos="1" side="tl" part="whole"/>
          </lu>
        </out>
      </action>
    </rule>

For reference, the pattern being matched are defined as:

  <section-def-cats>
    ...
    <def-cat n="v_neg_imp">
      <cat-item tags="v.*.neg.imp.*"/>
    </def-cat>

    ...
  </section-def-cats>

Description

So, given the description of the tags, and the contents of the rule, we could describe the rule as follows:

• This rule matches a sequence of "vb_neg_imp"
• The lexical form pattern "vb_neg_imp" is defined, in the <section-def-cats> of the file, as the tag <v> followed by any tag (e.g. <iv> or <tv> followed by the tags <neg><imp> and then any other tags.
• The rule then sets the value of the part of the lexical unit which matches the attribute a_neg to nothing (meaning it "deletes" it)
• The rule outputs the lemma "ан", followed by the tag <adv> in a single lexical unit (between ^ and $).
• The rule then outputs a blank space
• The rule then outputs the a lexical unit with the information from the verb matched by the original pattern.

Example output

Input text

$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob

^ye<v><tv><neg><imp><p2><sg>$^!<sent>$^.<sent>$

Output from transfer

$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob |\
   apertium-pretransfer | lt-proc -b tr-cv.autobil.bin | apertium-lrx-proc tr-cv.lrx.bin  |\
   apertium-transfer -b apertium-cv-tr.tr-cv.t1x tr-cv.t1x.bin 

^ан<adv>$ ^ҫи<v><tv><imp><p2><sg>$^!<sent>$

If you want to trace which rules are matched, add the -t option to the apertium-transfer:

Output from transfer

$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob |\
   apertium-pretransfer | lt-proc -b tr-cv.autobil.bin | apertium-lrx-proc tr-cv.lrx.bin  |\
   apertium-transfer -t -b apertium-cv-tr.tr-cv.t1x tr-cv.t1x.bin 

apertium-transfer: Rule 3 ye<v><tv><neg><imp><p2><sg>/ҫи<v><tv><neg><imp><p2><sg>
^ан<adv>$ ^ҫи<v><tv><imp><p2><sg>$^!<sent>$

Describe a new transfer rule

The objective of this part of the practical is to describe a new transfer rule. Run some text through the translator of your choice, and describe a new rule in natural language, as was done in the previous subsection. If you have time after defining the rule, try and write it out in XML format.

Further reading

• Ginestí i Rosell, M. (ed.) (2007) Documentation of the Open-Source Shallow-Transfer Machine Translation Platform Apertium