Курсы машинного перевода для языков России/Раздел 5
Цель данного раздела познакомить с понятием сопоставительного морфологического и синтаксического анализов, и показать как основные (т.е. that is, single level) правила морфологической и синтаксической передачи могут быть созданы в Apertium.
Contrastive analysis is the process of examining two or more languages together to find out what kind of features they share, and how they are distinguished. When working on shallow-transfer machine translation, we can consider for example, both morphological contrasts and syntactic contrasts.
Morphological contrasts are differences in which morphological features are expressed in each language, and how they differ. For example, if one language expresses definiteness morphologically, and another doesn't, or one language has case or gender while the other doesn't. Or if the case/gender inventories differ between the languages. Examples:
- Kazakh has no grammatical gender where Russian has three (masculine, feminine and neuter). Gender in Russian plays a rôle in agreement within noun phrases.
- Russian has two numbers (singular and plural), where Mansi has three (singular, dual and plural).
- Hungarian has a large number of cases which don't exist in Finnish (for example the temporal case -kor, which can be translated into Finnish by the ablative, adessive, or essive). Case in Hungarian and Finnish does not play a rôle in noun-phrase agreement.
Syntactic contrasts are differences in how the syntax of the languages works, for example the existence or not of articles, ordering of clitic pronouns, case inventories, analytic verb tenses, usage of subject pronouns, etc.
- Abkhaz has a postfixed indefinite article, and a prefixed definite article where Russian does not.
- Russian has compulsory subject pronouns, where Hungarian does not,
- Chuvash has an ablative case, where Russian uses a preposition and a different case.
Transfer is the process of altering the intermediate representation of a source language into that of the target language. In Apertium, transfer works on an intermediate representation of lemmas and tags specifying morphological features (lexical forms).
When thinking about implementing transfer rules, it is often worth trying to think in terms of some very basic operations.
- A) Insertion
Insertion is the operation of adding a new tag or word, for example to translate a noun from Dungan to Russian, a tag denoting case (nominative, accusative, ...) would need to be inserted, as Dungan does not have this morphological feature. Many times translation also requires the insertion of a word. For example when translating the imperative from Turkish to Chuvash, the adverb ан needs to be added to the beginning: Yeme! → Ан çи!.
- B) Deletion
Deletion is the opposite operation to insertion, a word or tag needs to be removed, for example as in the previous example, once the adverb has been inserted, the negative would need to be deleted (or not output) from the following verb, as it is the adverb which provides the negation.
- C) Substitution
Substitution is replacing one tag with another tag, such as for example changing the gender of a word. This is often done in the transfer lexicon, as we saw in the last session, but it can also take place in transfer. For example if you are translating from Russian to Bashkir, мои брюки, the word брюки would be marked with a special tag
ND (number to be determined) in the transfer lexicon, and then the transfer rules would substitute this with the appropriate number.
- D) Reordering
Reordering is changing the order of tags or words. For example the order of number and possessive when translating from Turkish into Chuvash kitap·lar·ım → кӗнеке·м·сем.
- E) Combining operations
A lot of the time, a combination of these operations is required to make a transfer rule, for example when translating from language with a case for a given meaning into a language with a preposition and a case for the same meaning, two operations are required, first the word for preposition needs to be inserted before the noun, and secondly the feature denoting case on the noun needs to be changed.
Left-to-right longest match (LRLM)
Files containing structural transfer rules in Apertium are laid out in the following fashion:
<transfer default="chunk"> <section-def-cats> <def-cat n="adj_or_pp"> <cat-item tags="adj.*"> <cat-item tags="vblex.pp.*"> </def-cat> ... </section-def-cats> <section-def-attrs> <def-attr n="nbr"> <attr-item n="sg"/> <attr-item n="pl"/> </def-attr> ... </section-def-attrs> <section-def-vars> <def-var n="number"/> ... </section-def-vars> <section-rules> ... </section-rules> </transfer>
This is the minimal layout. The four sections are:
<section-def-cats>: Contains one or more
<def-cat>entries. These specify patterns that can be matched by the transfer rules, and may match either tag sequences, or lemmas. The
<def-cat>entry above for
adj_or_ppmatches any lexical unit in which the tags start with
<vblex><pp>. This might be useful where two categories behave similarly (for example adjectives and past particiles in Spanish).
<section-def-attrs>: Contains one or more
<def-attr>entries. These list possible tags corresponding to a feature. For example, in this case the feature of
nbr"number" may be one of two tags
<pl>"plural". When we use the
<clip>tag, to extract a
part, this allows us to define our own parts, in addition to the default parts.
<section-def-vars>: Contains variable definitions. Variables are used to pass information between rules. For example we might want to keep track of the last gender or number that we have seen, or the lemma of the last finite verb.
<section-rules>: Contains the rules.
In addition to the parts that are defined in
section-def-attrs, the transfer module predefines a number of default parts of each lexical unit.
To more easily demonstrate the default parts that are available, we'll use the follow example:
^правило# по безопасности<n><nt><nn><pl>$
whole: as the name suggests, this provides the whole content of the lexical unit.
lem: the lemma: in this case, правило# по безопасности
tags: all of the tags, in one unit: <n><nt><nn><pl>
lemh: the lemma head; the part of the multiword which inflects: правило
lemq: the lemma queue; the rest of a multiword (the
<g></g>part in the dictionaries): # по безопасности
For this practical, we're going to use the Turkish--Chuvash language pair in Apertium, you could also choose another pair with single-level transfer (such as Tatar--Bashkir). So, go to the directory
apertium-cv-tr and open the file of transfer rules for Turkish to Chuvash
apertium-cv-tr.tr-cv.t1x, in other pairs this might be a
Describe a transfer rule
The objective of this exercise is to describe the behaviour of an existing transfer rule given the input sentence in Turkish: Yeme!. First we're going to give some preliminaries for how transfer rules are structured, then we're going to look at an actual rule and describe it.
Transfer rules in Apertium are made up of two principle parts. A pattern — that is a sequence of categories which will be matched, and an action, which contains operations that are carried out on the patterns matched. An overall schema for a rule might look like the following:
<rule> <pattern> ... </pattern> <action> ... </action> </rule>
An overview of the meanings of the tags which are used in the example below are given here, along with references to the page of the documentation where a full description can be found.
||§126.96.36.199||Starts a new rule, it contains at the highest level a pattern and an action.|
||§188.8.131.52||Contains one or more |
||§184.108.40.206||Contains references to patterns of lexical forms defined in the |
||§220.127.116.11||Contains tags defining the actions that should take place when a sequence of lexical units matching |
||§18.104.22.168||Contains one or more |
||§22.214.171.124||Specifies a condition and an action to take if that condition is fulfilled. The condition is in a |
||§126.96.36.199||A default condition, which contains actions that are run if none of the |
||§188.8.131.52||Describes a condition. For example "test if the number of the first pattern matched is singular".|
||§184.108.40.206||A tag allowing more than one condition to be matched. This tag is a Boolean operator, other Boolean operators available are: |
||§220.127.116.11||Tests if two strings or tags are equal.|
||§18.104.22.168||Extracts the part of a lexical unit corresponding to an attribute as defined in the |
||§22.214.171.124||Generates a string enclosed in |
||§126.96.36.199||Generates a string.|
||§188.8.131.52||Contains everything that will be output by the rule.|
||§184.108.40.206||Encloses the contents in |
||§220.127.116.11||Outputs a space, or the formatting contained at the given position.|
||§18.104.22.168||Converts the case (e.g. upper, lower) of whatever is enclosed in it to the case of the word marked by the |
The rule we're going to describe is
regla: v_neg_imp, so search for it in the file. Here is what the rule looks like:
<rule comment="regla: v_neg_imp (yeme! → ан ҫи!)"> <pattern> <pattern-item n="v_neg_imp"/> </pattern> <action> <let> <clip pos="1" side="tl" part="a_neg"/> <lit v=""/> </let> <out> <lu> <lit v="ан"/> <lit-tag v="adv"/> </lu> <b/> <lu> <clip pos="1" side="tl" part="whole"/> </lu> </out> </action> </rule>
For reference, the pattern being matched are defined as:
<section-def-cats> ... <def-cat n="v_neg_imp"> <cat-item tags="v.*.neg.imp.*"/> </def-cat> ... </section-def-cats>
So, given the description of the tags, and the contents of the rule, we could describe the rule as follows:
- This rule matches a sequence of "vb_neg_imp"
- The lexical form pattern "vb_neg_imp" is defined, in the
<section-def-cats>of the file, as the tag
<v>followed by any tag (e.g.
<tv>followed by the tags
<neg><imp>and then any other tags.
- The rule then sets the value of the part of the lexical unit which matches the attribute
a_negto nothing (meaning it "deletes" it)
- The rule outputs the lemma "ан", followed by the tag
<adv>in a single lexical unit (between
- The rule then outputs a blank space
- The rule then outputs the a lexical unit with the information from the verb matched by the original pattern.
- Input text
$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob ^ye<v><tv><neg><imp><p2><sg>$^!<sent>$^.<sent>$
- Output from transfer
$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob |\ apertium-pretransfer | lt-proc -b tr-cv.autobil.bin | apertium-lrx-proc tr-cv.lrx.bin |\ apertium-transfer -b apertium-cv-tr.tr-cv.t1x tr-cv.t1x.bin ^ан<adv>$ ^ҫи<v><tv><imp><p2><sg>$^!<sent>$
If you want to trace which rules are matched, add the -t option to the apertium-transfer:
- Output from transfer
$ echo 'yeme!' | hfst-proc tr-cv.automorf.hfst | cg-proc tr-cv.rlx.bin | apertium-tagger -g tr-cv.prob |\ apertium-pretransfer | lt-proc -b tr-cv.autobil.bin | apertium-lrx-proc tr-cv.lrx.bin |\ apertium-transfer -t -b apertium-cv-tr.tr-cv.t1x tr-cv.t1x.bin apertium-transfer: Rule 3 ye<v><tv><neg><imp><p2><sg>/ҫи<v><tv><neg><imp><p2><sg> ^ан<adv>$ ^ҫи<v><tv><imp><p2><sg>$^!<sent>$
Describe a new transfer rule
The objective of this part of the practical is to describe a new transfer rule. Run some text through the translator of your choice, and describe a new rule in natural language, as was done in the previous subsection. If you have time after defining the rule, try and write it out in XML format.
- Ginestí i Rosell, M. (ed.) (2007) Documentation of the Open-Source Shallow-Transfer Machine Translation Platform Apertium