Uputstvo za novi jezički par za Apertium

Uputstvo za novi jezički par za Apertium

Ovo uputstvo će objasniti kako započeti novi jezički par za Apertium mašinski prevod. Ne podrazumevamo poznavanje lingvistine ili mašinskog prevoda - dovoljno je da znati da razlikujete različite vrste reči (imenice, glagole, prideve itd.)

Reč-dve o srpskohrvatskom prevodu

Ovaj prevod koristi Unicode prikaz slova Lj, Nj i Dž umesto Lj, Nj i Dž. Ukoliko ne vidite ta slova, instalirajte set slobodnih fontova koji podržavaju Unicode, kao što su DejaVu fontovi i postarajte se da gledate ovu stranicu u UTF8 kodnom rasporedu.

U prevodu takođe koristimo neke neologizme koji su se ustalili u upotrebi na srpskohrvatskom govornom području. Npr. file „prevodimo“ kao fajl umesto uobičajenog datoteka.

Uvod

Kao što ste možda primetili iz uvoda, Apertium je sistema mašinskog prevođenja. Preciznije, u pitanju je platforma za prevod koja obuhvata pokretački motor i alatke koje dozvoljavaju pravljenje ličnog sistema mašinskog prevođenja. Vi samo treba da unesete podatke. U osnovi, podaci se sastoje iz tri rečnika i nekoliko pravila koji se staraju za tačan red reči u rečenici i druge gramatičke zavrzlame.

Za detaljnije uputstvo o tome kako ovo sve radi, pročitajte dokumentaciju na sajtu projekta na apertium.sourceforge.net.

Trebaće Vam

• lttoolbox (>= 3.0.0)
• libxml utils (xmllint itd.)
• apertium (>= 3.0.0)
• editor teksta (ili specijalizovani uređivač za XML, ako Vam tako više odgovara)

Za instrukcije kako da instalirate ove programe pogledajte dokumentaciju na sajtu Apertiuma.

Od čega se sastoji jezički par?

Apertium koristi „plitki“ mašinski prevod, što znači da se koristi rečnicima i plitkim pravilima prenosa.

„Plitki prevod“ se razlikuje od „dubokog prevoda“ utoliko što se ne bavi punom sintatičkom obradom. Pravila se obično primenjuju na grupe leksičkih jedinica, umesto na razgranatu obradu. U osnovi postoje tri glavna rečnika.

1. Morfološki rečnik za jezik xx: on sadrži pravila o tome kako se menjaju reči u jeziku xx. U našem primeru ovo ćemo zvati: apertium-sh-en.sh.dix.
2. Morfološki rečnik za jezik yy: on sadrži pravila o tome kako se menjaju reči u jeziku yy. U našem primeru ovo ćemo zvati: apertium-sh-en.en.dix.
3. Dvojezički rečnik: on sadrži odnos između reči i simbola dva jezika. U našem primeru ovo ćemo zvati: apertium-sh-en.sh-en.dix.

U paru za prevod, oba jezika mogu biti i izvor i cilj prevoda, te su ovi izrazi relativni.

Postoje i dva fajla za pravila prevoda i to su pravila o redu reči u rečenici, npr. chat noir → cat black → black cat. Takođe se staraju i o slaganju roda i broja i sl. Pravila se mogu koristiti i za ubacivanje i prisanje leksičkih stavki, a o tome ćemo kasnije. Fajlovi u pitanju su:

• pravila prevoda jezika xx u jezik yy: ovaj fajl sadrži pravila po kojim će se jezik xx menjati u jezik yy. U našem primeru ovo ćemo zvati: apertium-sh-en.trules-sh-en.xml
• pravila prevoda jezika yy u jezik xx: ovaj fajl sadrži pravila po kojim će se jezik yy menjati u jezik xx. U našem primeru ovo ćemo zvati: apertium-sh-en.trules-sh-en.xml

Iako u postojećim jezičkim parovima postoje i neki drugi fajlovi, ovi koje smo naveli su neophodni i dovoljni za funkcionalni sistem.

Jezički par

Možda ste već skapirali iz imena fajlova, ali ovo uputstvo će se koristiti prevod srpskohrvatskog u engleski radi objašnjenja kako da napravite osnovni sistem. Primer nije idealan, jer sistem bolje funkcioniše na srodnijim jezicima.

O terminologiji

Pre nego što nastavimo, bitno je da pojasnimo neke izraze:

Prvi izraz je lema. U pitanju je izvorni oblik reči, kakav se nalazi u rečniku. Recimo, nominativ jednine za imenice, te infinitiv za glagole. Glagol „raditi“ je lema, dok je „radim“ ili „radiš“ njegova inflikcija. Reč „mačke“ je inflikcija leme „mačka“

Drugi izraz je simbol. U našem kontekstvu, simbol je gramatička oznaka. Reč „mačke“ je množina imenice, te će imati simbole imenice i množine. Ova informacija se obično stavlja u kose zagrade, recimo:

• <n>; za imenicu (od španskog nom)
• <pl>; za množinu (od španskog plural).

Neki drugi simboli su <sg> jednina, <p1> prvo lice itd. Valja napomenuti da su u većini jezičkih parova korišćeni katalonski izrazi za simbole. Npr. vbhaver - od vb (verb) i haver (imati). Simboli su definisani u <sdef> tagovima i korišćeni u <s> tagovima.

Treća reč je paradigma. U našem kontekstu, paradigma označava primer kako se neka grupa reči menja. U morfološkim rečnicima leme su povezane s paradigmama koje nam dozvoljavaju da opišemo kako se data reč menja bez pisanja svakog pojedinačnog nastavka.

Recimo da želimo da opišemo komparaciju prideva „srećan“ i „dosadan“:

• srećan, sreć (an, niji, naj - niji)
• dosadan, dosad (an, niji, naj - niji)

Dovoljno je da napišemo jedno pravilo za pridev koje se primenjuje na mnogo drugih prideva. Paradigme se definišu u tagu <pardef>, a koriste u <par>.

Sâm početak

Jednojezički rečnici

Prvo ćemo napraviti rečnik za izvorni jezik, koji se nalazi u XML fajlu. Pokrenite tekst editor i ukucajte sledeće: and type the following:

<?xml version="1.0" encoding="UTF-8"?>
<dictionary>

</dictionary>

Dakle, fajl do sada definiše našu želju da sačinimo novi rečnik. Da bi taj rečnik bio iole koristan, neophodno je da prvo dodamo abecedu - spisak slova koje ćemo koristiti u rečniku. Za srpskohrvatski:

<alphabet>ABCČĆDDžĐEFGHIJKLLjMNNjOPRSŠTUVZŽabcčćddžđefghijklljmnnjoprsštuvzž</alphabet>

Dodajte abecedu ispod <dictionary> taga.

Sada treba da definišemo neke simbole. Počnimo s jednostavnim - imenica (i) u jednini (j) ili množini (m).

<sdefs>
   <sdef n="i"/>
   <sdef n="j"/>
   <sdef n="m"/>
</sdefs>

Simboli ne moraju biti kratki, možete koristiti i ce izraz ukoliko želite. Ipak, pošto ćete to često pisati, ima smisla koristiti skraćenice.

Očigledno, nije sve tako jednostavno jer se reči deklinuju i konjuguju itd. ali ćemo za potrebe ovog primera pretpostaviti da je imenica muškog roda u jednini.

Sledeći zadatak je da definišemo paradigme:

<pardefs>

</pardefs>

i sekciju za rečnik:

<section id="main" type="standard">

</section>

Postoje dve vrste sekcija - prva je standardna, koja sadrži reči, enklitike itd. Drugi tip sadrži stvari kao što interpunkcija i ostale nezavisne simbole. Ovde tu sekciju nemamo, ali ćemo je kasnije demonstrirati.

Konačno, naš fajl će izgledati ovako:

<?xml version="1.0" encoding="UTF-8"?>
<dictionary>
   <sdefs>
     <sdef n="n"/>
     <sdef n="sg"/>
     <sdef n="pl"/>
   </sdefs>
   <pardefs>

   </pardefs>
   <section id="main" type="standard">

   </section>
</dictionary>

Pošto smo postavili temelj rečniku, možemo dodati imenicu. Igrom slučaja, izabrali smo da za demonstraciju koristimo imenicu „gramofon“.

The first thing we need to do, as we have no prior paradigms, is to define a paradigm.

Remember we're assuming masculine gender and nominative case. The singular form of the noun is 'gramofon', and the plural is 'gramofoni'. So:

<pardef n="gramofon__n">
   <e>
     <p>
       <l/>
       <r><s n="n"/><s n="sg"/></r>
     </p>
   </e>
   <e>
     <p>
       <l>i</l>
       <r><s n="n"/><s n="pl"/></r>
     </p>
   </e>
</pardef>

Note: the '<l/>' (equivalent to <l></l>) denotes that there is no extra material to be added to the stem for the singular.

This may seem like a rather verbose way of describing it, but there are reasons for it and it quickly becomes second nature. You're probably wondering what the <e>,

, <l> and <r> stand for. Well,

• e, is for entry.
• p, is for pair.
• l, is for left.
• r, is for right.

Why left and right? Well, the morphological dictionaries will later be compiled into finite state machines. Compiling them left to right produces analyses from words, and from right to left produces words from analyses. For example:

* gramofoni (left to right) gramofon<n><pl> (analysis)
* gramofon<n><pl> (right to left) gramofoni (generation)

Now we've defined a paradigm, we need to link it to its lemma, gramofon. We put this in the section that we've defined.

The entry to put in will look like:

<e lm="gramofon"><i>gramofon</i><par n="gramofon__n"/></e>

A quick run down on the abbreviations:

• lm, is for lemma.
• i, is for identity (the left and the right are the same).
• par, is for paradigm.

This entry states the lemma of the word, gramofon, the root, gramofon and the paradigm with which it inflects gramofon__n. The difference between the lemma and the root is that the lemma is the citation form of the word, while the root is the substring of the lemma to which stems are added. This will become clearer later when we show an entry where the two are different.

We're now ready to test the dictionary. Save it, and then return to the shell. We first need to compile it (with lt-comp), then we can test it (with lt-proc).

$ lt-comp lr apertium-sh-en.sh.dix sh-en.automorf.bin

Should produce the output:

main@standard 12 12

As we are compiling it left to right, we're producing an analyser. Lets make a generator too.

$ lt-comp rl apertium-sh-en.sh.dix sh-en.autogen.bin

At this stage, the command should produce the same output.

We can now test these. Run lt-proc on the analyser.

$ lt-proc sh-en.automorf.bin

Now try it out, type in gramofoni (gramophones), and see the output:

^gramofoni/gramofon<n><pl>$

Now, for the English dictionary, do the same thing, but substitute the English word gramophone for gramofon, and change the plural inflection. What if you want to use the more correct word 'record player', well, we'll explain how to do that later.

You should now have two files in the directory:

• apertium-sh-en.sh.dix which contains a (very) basic Serbo-Croatian morphological dictionary, and
• apertium-sh-en.en.dix which contains a (very) basic English morphological dictionary.

Bilingual dictionary

So we now have two morphological dictionaries, next thing to make is the bilingual dictionary. This describes mappings between words. All dictionaries use the same format (which is specified in the DTD, dix.dtd).

Create a new file, apertium-sh-en.sh-en.dix and add the basic skeleton:

<?xml version="1.0" encoding="UTF-8"?>
<dictionary>
   <alphabet/>
   <sdefs>
     <sdef n="n"/>
     <sdef n="sg"/>
     <sdef n="pl"/>
   </sdefs>

   <section id="main" type="standard">

   </section>
</dictionary>

Now we need to add an entry to translate between the two words. Something like:

<e><p><l>gramofon<s n="n"/></l><r>gramophone<s n="n"/></r></p></e>

Because there are a lot of these entries, they're typically written on one line to facilitate easier reading of the file. Again with the 'l' and 'r' right? Well, we compile it left to right to produce the Serbo-Croatian → English dictionary, and right to left to produce the English → Serbo-Croatian dictionary.

So, once this is done, run the following commands:

$ lt-comp lr apertium-sh-en.sh.dix sh-en.automorf.bin
$ lt-comp rl apertium-sh-en.sh.dix sh-en.autogen.bin

$ lt-comp lr apertium-sh-en.en.dix en-sh.automorf.bin
$ lt-comp rl apertium-sh-en.en.dix en-sh.autogen.bin

$ lt-comp lr apertium-sh-en.sh-en.dix sh-en.autobil.bin
$ lt-comp rl apertium-sh-en.sh-en.dix en-sh.autobil.bin

To generate the morphological analysers (automorf), the morphological generators (autogen) and the word lookups (autobil), the bil is for "bilingual".

Transfer rules

So, now we have two morphological dictionaries, and a bilingual dictionary. All that we need now is a transfer rule for nouns. Transfer rule files have their own DTD (transfer.dtd) which can be found in the Apertium package. If you need to implement a rule it is often a good idea to look in the rule files of other language pairs first. Many rules can be recycled/reused between languages. For example the one described below would be useful for any null-subject language.

Start out like all the others with a basic skeleton:

<?xml version="1.0" encoding="UTF-8"?>
<transfer>

</transfer>

At the moment, because we're ignoring case, we just need to make a rule that takes the grammatical symbols input and outputs them again.

We first need to define categories and attributes. Categories and attributes both allow us to group grammatical symbols. Categories allow us to group symbols for the purposes of matching (for example 'n.*' is all nouns). Attributes allow us to group a set of symbols that can be chosen from. For example ('sg' and 'pl' may be grouped a an attribute 'number').

Lets add the necessary sections:

<section-def-cats>

</section-def-cats>
<section-def-attrs>

</section-def-attrs>

As we're only inflecting, nouns in singular and plural then we need to add a category for nouns, and with an attribute of number. Something like the following will suffice:

Into section-def-cats add:

<def-cat n="nom">
   <cat-item tags="n.*"/>
</def-cat>

This catches all nouns (lemmas followed by <n> then anything) and refers to them as "nom" (we'll see how thats used later).

Into the section section-def-attrs, add:

<def-attr n="nbr">
   <attr-item tags="sg"/>
   <attr-item tags="pl"/>
</def-attr>

and then

<def-attr n="a_nom">
   <attr-item tags="n"/>
</def-attr>

The first defines the attribute nbr (number), which can be either singular (sg) or plural (pl).

The second defines the attribute a_nom (attribute noun).

Next we need to add a section for global variables:

<section-def-vars>

</section-def-vars>

These variables are used to store or transfer attributes between rules. We need only one for now,

<def-var n="number"/>

Finally, we need to add a rule, to take in the noun and then output it in the correct form. We'll need a rules section...

<section-rules>

</section-rules>

Changing the pace from the previous examples, I'll just paste this rule, then go through it, rather than the other way round.

<rule>
   <pattern>
     <pattern-item n="nom"/>
   </pattern>
   <action>
     <out>
       <lu>
         <clip pos="1" side="tl" part="lem"/>
         <clip pos="1" side="tl" part="a_nom"/>
         <clip pos="1" side="tl" part="nbr"/>
       </lu>
     </out>
   </action>
</rule>

The first tag is obvious, it defines a rule. The second tag, pattern basically says: "apply this rule, if this pattern is found". In this example the pattern consists of a single noun (defined by the category item nom). Note that patterns are matched in a longest-match first. So if you have three rules, the first catches "<prn><vblex><n>", the second catches "<prn><vblex>" and the third catches "<n>", the pattern matched, and rule executed will be the first.

For each pattern, there is an associated action, which produces an associated output, out. The output, is a lexical unit (lu).

The clip tag allows a user to select and manipulate attributes and parts of the source language (side="sl"), or target language (side="tl") lexical item.

Let's compile it and test it. Transfer rules are compiled with:

$ apertium-preprocess-transfer apertium-sh-en.trules-sh-en.xml trules-sh-en.bin

Which will generate a trules-sh-en.bin file.

Now we're ready to test our machine translation system. There is one crucial part missing, the part-of-speech (PoS) tagger, but that will be explained shortly. In the meantime we can test it as is:

First, lets analyse a word, gramofoni:

$ echo "gramofoni" | lt-proc sh-en.automorf.bin 
^gramofon/gramofon<n><pl>$

Now, normally here the POS tagger would choose the right version based on the part of speech, but we don't have a POS tagger yet, so we can use this little gawk script (thanks to Sergio) that will just output the first item retrieved.

$ echo "gramofoni" | lt-proc sh-en.automorf.bin | \
  gawk 'BEGIN{RS="$"; FS="/";}{nf=split($1,COMPONENTS,"^"); for(i = 1; i<nf; i++) printf COMPONENTS[i]; if($2 != "") printf("^%s$",$2);}' | \
^gramofon<n><pl>$

Now let's process that with the transfer rule:

$ echo "gramofoni" | lt-proc sh-en.automorf.bin | \
  gawk 'BEGIN{RS="$"; FS="/";}{nf=split($1,COMPONENTS,"^"); for(i = 1; i<nf; i++) printf COMPONENTS[i]; if($2 != "") printf("^%s$",$2);}' | \
  apertium-transfer apertium-sh-en.trules-sh-en.xml trules-sh-en.bin sh-en.autobil.bin

It will output:

^gramophone<n><pl>$^@

• 'gramophone' is the target language (side="tl") lemma (lem) at position 1 (pos="1").
• '<n>' is the target language a_nom at position 1.
• '<pl>' is the target language attribute of number (nbr) at position 1.

Try commenting out one of these clip statements, recompiling and seeing what happens.

So, now we have the output from the transfer, the only thing that remains is to generate the target-language inflected forms. For this, we use lt-proc, but in generation (-g), not analysis mode.

$ echo "gramofoni" | lt-proc sh-en.automorf.bin | \
  gawk 'BEGIN{RS="$"; FS="/";}{nf=split($1,COMPONENTS,"^"); for(i = 1; i<nf; i++) printf COMPONENTS[i]; if($2 != "") printf("^%s$",$2);}' | \
  apertium-transfer apertium-sh-en.trules-sh-en.xml trules-sh-en.bin sh-en.autobil.bin | \
  lt-proc -g sh-en.autogen.bin

gramophones\@

And c'est ca. You now have a machine translation system that translates a Serbo-Croatian noun into an English noun. Obviously this isn't very useful, but we'll get onto the more complex stuff soon. Oh, and don't worry about the '@' symbol, I'll explain that soon too.

Think of a few other words that inflect the same as gramofon. How about adding those. We don't need to add any paradigms, just the entries in the main section of the monolingual and bilingual dictionaries.