Difference between revisions of "ReTraTos"

From Apertium
Jump to navigation Jump to search
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[L'outil ReTraTos|En français]]

{{TOCD}}
'''ReTraTos''' is a toolbox to build linguistic resources useful for machine translation (MT): bilingual dictionaries and transfer rules. The induction systems and open linguistic data can be used with the [[Apertium]] toolbox to build open-source MT systems.

==Bilingual dictionaries==

This section describes how to use ReTraTos to create a bilingual dictionary for your Apertium language pair. You will need:

* An aligned [[corpus]] of the two languages. For any pair of european languages, the JRC-Acquis corpus is recommended.
* apertium
* [[lttoolbox]]
* ReTraTos
* [[GIZA++]]
* a lot of patience

===Preparing the corpus===

The corpus should be in two files, each with one sentence per line. For example,

;es.txt
<pre>
Reconociendo que , en particular , sería mutuamente beneficioso cooperar mediante el establecimiento de un programa común de investigaciones y de
desarrollo ;
Considerando que un acuerdo que establezca una cooperación en el ámbito de las utilizaciones pacíficas de la energía atómica iniciaría fructíferos
intercambios de experiencia
...
</pre>
;it.txt
<pre>
Riconoscendo in particolare che sarebbe loro reciproco vantaggio cooperare con lo stabilire un programma comune di ricerche e di
sviluppo ;
Considerando che un accordo inteso a stabilire una cooperazione nel campo degli usi pacifici dell'energia atomica darebbe inizio ad un proficuo
scambio di esperienze
...
</pre>

Depending on if you want to create entries for proper names, you could lower-case the whole corpus. Make sure that there are spaces in between any punctuation characters, otherwise the punctuation will be counted as part of the word.

===Tagging the corpus===

Tag both of the files using the apertium-tagger. So, for example for Spanish:

<pre>
$ cat es.txt | apertium-destxt | lt-proc es-it.automorf.bin | apertium-tagger -g es-it.prob | apertium-retxt > es.tagged.txt &
</pre>

After this, strip out the '^' and the '$' symbols from the es-tagged.txt file. This will result in lines that look something like:

<pre>
Reconocer<vblex><ger> que<cnjsub> ,<cm> en<pr> particular<adj><mf><sg> ,<cm> ser<vbser><cni><p3><sg> mutuamente<adv>
beneficioso<adj><m><sg> *cooperar mediante<pr> el<det><def><m><sg> establecimiento<n><m><sg> de<pr>
uno<det><ind><m><sg> programa<n><m><sg> común<adj><mf><sg> de<pr> investigación<n><f><pl> y<cnjcoo>
de<pr> desarrollo<n><m><sg> ;<sent>
</pre>


ReTraTos is very picky about formatting, so before removing '^' and '$' you might want to just remove anything that's ''not'' contained within '^' and '$', ie. something like
<pre>
# ^foo$-^bar$ => ^foo$ ^bar$ -^foo$ => ^foo$ ^bar$- => ^bar$
sed 's%\$[^^]*\^%\$ \^%g' | sed 's%^[^^]*\^%\^%g' | sed 's%\$[^^]*$%\$%g' |\
# remove '^' and '$'
sed 's%[$^]% %g'
</pre>

This'll save you lots of time later on... oh, and you should also just remove any lines that don't have at least one <code>^foo<bar>$</code> sequence in them (eg. a single alignment of '%' to '%', or something).

===Aligning the corpus===
{{see-also|Using GIZA++|Getting started with induction tools}}

Use GIZA++ to align both of the files, the instructions can be found in the page [[using GIZA++]]. Once the alignment has been made, you will end up with a file that ends in <code>.A3.final</code>, this is your alignment file. Next you will need to convert the alignment to the LIHLA format that ReTraTos uses. The script on the [[Talk:ReTraTos|talk page]] serves this purpose. For Spanish--Italian, it would be called thusly:

<pre>
$ perl giza_to_lihla.pl es_it.aligned.A3.final ./es/ ./it/
</pre>

This will put two files into the directories <code>./es/</code> and <code>./it/</code> which correspond to the lines in Spanish and Italian respectively. These LIHLA alignment files will end in <code>.al</code> and will look like the following:

<pre>
<s snum=6>Reconocer<vblex><ger>:1 que<cnjsub>:0 ,<cm>:0 en<pr>:2 particular<adj><mf><sg>:3 ,<cm>:0 ser<vbser><cni><p3><sg>:5 mutuamente<adv>:7
beneficioso<adj><m><sg>:8 *cooperar:9 mediante<pr>:10 el<det><def><m><sg>:11 establecimiento<n><m><sg>:12 de<pr>:0 uno<det><ind><m><sg>:13
programa<n><m><sg>:14 común<adj><mf><sg>:15 de<pr>:16 investigación<n><f><pl>:17 y<cnjcoo>:18 de<pr>:19 desarrollo<n><m><sg>:20 ;<sent>:0</s>
</pre>

In order to use ReTraTos, the surface forms of each lexical unit need to be re-added so that the above example looks like the following:

<pre>
<s snum=6>Reconociendo/Reconocer<vblex><ger>:1 que/que<cnjsub>:0 ,/,<cm>:0 en/en<pr>:2 particular/particular<adj><mf><sg>:3 ,/,<cm>:0
sería/ser<vbser><cni><p3><sg>:5 mutuamente/mutuamente<adv>:7 beneficioso/beneficioso<adj><m><sg>:8 *cooperar/*cooperar:9 mediante/mediante<pr>:10
el/el<det><def><m><sg>:11 establecimiento/establecimiento<n><m><sg>:12 de/de<pr>:0 un/uno<det><ind><m><sg>:13 programa/programa<n><m><sg>:14
común/común<adj><mf><sg>:15 de/de<pr>:16 investigaciones/investigación<n><f><pl>:17 y/y<cnjcoo>:18 de/de<pr>:19 desarollo/desarrollo<n><m><sg>:20
;<sent>:0</s>
</pre>

===Running ReTraTos_lex===

You will need the header and footer of a bilingual dictionary in two separate files, for example, <code>dic_header.txt</code> and <code>dic_footer.txt</code> (see the examples in the package). Then the program for generating the dictionary (<code>ReTraTos_lex</code>) can be called like this:

<pre>
$ ReTraTos_lex -s ./es/es_it.aligned.A3.final.al -t ./it/es_it.aligned.A3.final.al -b dic_header.txt -e dic_footer.txt

PRE-PROCESSAMENTO

Reading the examples ... 100000 examples read
Reading the examples ... 100000 examples read

GERANDO LEXICO

Generating source-target dictionary ... OK
Generating target-source dictionary ... OK
Processing bilingual dictionary ... OK
Generalising bilingual dictionary ... OK
Cleaning equal attributes ... OK

IMPRIMINDO LEXICO

Printing bilingual dictionary ... OK
</pre>

The output file will be the <code>.dix</code> file.

===Troubleshooting===

;Wrongly tagged

If you are getting many messages like this, or even just one, it means that there is an error in one of your alignment files.

<pre>
WARNING: (Entrada::le_sentenca): String O<cnjcoo>.<sent>V<num><mf><pl> is wrongly tagged
WARNING: (Entrada::le_sentenca): String )<rpar>.<sent> is wrongly tagged
WARNING: (Entrada::le_sentenca): String a<pr>)<rpar> is wrongly tagged
WARNING: (Entrada::le_sentenca): String 1991<num>.<sent> is wrongly tagged

...
</pre>

The alignments in the files must be in the format of <code>lemma<tags>:<alignment></code>, e.g. <code>1991<num>:1</code> or <code>pero<cnjcoo>:2</code>. Forbidden are the alignments which have random strings between tags, e.g. <code>O<cnjcoo>.<sent>V<num><mf><pl></code> is a set of three lemma/tag pairs with the same alignment. Probably the punctuation has not been separated out correctly from the text with spaces.

It is possible to fix this manually, by going in and either removing punctuation, or inserting spaces and null-alignments, but it might be better if there are many warning messages to re-process the corpus.

==See also==

* [[Using GIZA++]]
* [[Getting started with induction tools]]

==External links==
==External links==
* [http://retratos.sourceforge.net/ ReTraTos: Homepage]
* [http://www.nilc.icmc.usp.br/nilc/projects/retratos.htm ReTraTos: Homepage]
* [http://retratos.svn.sourceforge.net/viewvc/retratos/ ReTraTos] (View SVN)
* [http://retratos.svn.sourceforge.net/viewvc/retratos/ ReTraTos: SVN]
* [https://github.com/apertium/ReTraTos ReTraTos: github archive of the SVN]


==Further reading==
==Further reading==


* Helena M. Caseli, Maria das Graças V. Nunes, Mikel L. Forcada, "[http://www.dlsi.ua.es/~mlf/docum/caseli08p.pdf From free shallow monolingual resources to machine translation systems: easing the task]", in ''Mixing Approaches To Machine Translation'', MATMT2008, proceedings (Donostia, Spain, Feb. 14, 2008), pp. 41--48
* Helena M. Caseli, Maria das Graças V. Nunes, Mikel L. Forcada. (2008) "[https://www.dlsi.ua.es/gent/mlf/docum/caseli08.pdf From free shallow monolingual resources to machine translation systems: easing the task]", in ''Mixing Approaches To Machine Translation'', MATMT2008, proceedings (Donostia, Spain, Feb. 14, 2008), pp. 41--48
* Helena M. Caseli, Maria das Graças V. Nunes, Mikel L. Forcada. (2008) "[https://www.dlsi.ua.es/gent/mlf/docum/caseli08j.pdf Automatic induction of bilingual resources from aligned parallel corpora: application to shallow-transfer machine translation]". ''Machine Translation''




[[Category:Tools]]
[[Category:Tools]]
[[Category:Documentation in English]]

Latest revision as of 09:32, 3 May 2024

En français

ReTraTos is a toolbox to build linguistic resources useful for machine translation (MT): bilingual dictionaries and transfer rules. The induction systems and open linguistic data can be used with the Apertium toolbox to build open-source MT systems.

Bilingual dictionaries[edit]

This section describes how to use ReTraTos to create a bilingual dictionary for your Apertium language pair. You will need:

  • An aligned corpus of the two languages. For any pair of european languages, the JRC-Acquis corpus is recommended.
  • apertium
  • lttoolbox
  • ReTraTos
  • GIZA++
  • a lot of patience

Preparing the corpus[edit]

The corpus should be in two files, each with one sentence per line. For example,

es.txt
Reconociendo que , en particular , sería mutuamente beneficioso cooperar mediante el establecimiento de un programa común de investigaciones y de 
desarrollo ;
Considerando que un acuerdo que establezca una cooperación en el ámbito de las utilizaciones pacíficas de la energía atómica iniciaría fructíferos 
intercambios de experiencia 
...
it.txt
Riconoscendo in particolare che sarebbe loro reciproco vantaggio cooperare con lo stabilire un programma comune di ricerche e di 
sviluppo ;
Considerando che un accordo inteso a stabilire una cooperazione nel campo degli usi pacifici dell'energia atomica darebbe inizio ad un proficuo 
scambio di esperienze
...

Depending on if you want to create entries for proper names, you could lower-case the whole corpus. Make sure that there are spaces in between any punctuation characters, otherwise the punctuation will be counted as part of the word.

Tagging the corpus[edit]

Tag both of the files using the apertium-tagger. So, for example for Spanish: 

$ cat es.txt | apertium-destxt | lt-proc es-it.automorf.bin | apertium-tagger -g es-it.prob | apertium-retxt > es.tagged.txt &

After this, strip out the '^' and the '$' symbols from the es-tagged.txt file. This will result in lines that look something like: 

Reconocer<vblex><ger> que<cnjsub> ,<cm> en<pr> particular<adj><mf><sg> ,<cm> ser<vbser><cni><p3><sg> mutuamente<adv> 
beneficioso<adj><m><sg> *cooperar mediante<pr> el<det><def><m><sg> establecimiento<n><m><sg> de<pr> 
uno<det><ind><m><sg> programa<n><m><sg> común<adj><mf><sg> de<pr> investigación<n><f><pl> y<cnjcoo> 
de<pr> desarrollo<n><m><sg> ;<sent>


ReTraTos is very picky about formatting, so before removing '^' and '$' you might want to just remove anything that's not contained within '^' and '$', ie. something like 

# ^foo$-^bar$ => ^foo$ ^bar$     -^foo$ => ^foo$           ^bar$- => ^bar$
sed 's%\$[^^]*\^%\$ \^%g'    | sed  's%^[^^]*\^%\^%g' | sed 's%\$[^^]*$%\$%g' |\
# remove '^' and '$'
sed 's%[$^]% %g'

This'll save you lots of time later on... oh, and you should also just remove any lines that don't have at least one ^foo<bar>$ sequence in them (eg. a single alignment of '%' to '%', or something).

Aligning the corpus[edit]

See also: Using GIZA++ and Getting started with induction tools

Use GIZA++ to align both of the files, the instructions can be found in the page using GIZA++. Once the alignment has been made, you will end up with a file that ends in .A3.final, this is your alignment file. Next you will need to convert the alignment to the LIHLA format that ReTraTos uses. The script on the talk page serves this purpose. For Spanish--Italian, it would be called thusly: 

$ perl giza_to_lihla.pl es_it.aligned.A3.final ./es/ ./it/

This will put two files into the directories ./es/ and ./it/ which correspond to the lines in Spanish and Italian respectively. These LIHLA alignment files will end in .al and will look like the following: 

<s snum=6>Reconocer<vblex><ger>:1 que<cnjsub>:0 ,<cm>:0 en<pr>:2 particular<adj><mf><sg>:3 ,<cm>:0 ser<vbser><cni><p3><sg>:5 mutuamente<adv>:7 
beneficioso<adj><m><sg>:8 *cooperar:9 mediante<pr>:10 el<det><def><m><sg>:11 establecimiento<n><m><sg>:12 de<pr>:0 uno<det><ind><m><sg>:13 
programa<n><m><sg>:14 común<adj><mf><sg>:15 de<pr>:16 investigación<n><f><pl>:17 y<cnjcoo>:18 de<pr>:19 desarrollo<n><m><sg>:20 ;<sent>:0</s>

In order to use ReTraTos, the surface forms of each lexical unit need to be re-added so that the above example looks like the following: 

<s snum=6>Reconociendo/Reconocer<vblex><ger>:1 que/que<cnjsub>:0 ,/,<cm>:0 en/en<pr>:2 particular/particular<adj><mf><sg>:3 ,/,<cm>:0 
sería/ser<vbser><cni><p3><sg>:5 mutuamente/mutuamente<adv>:7 beneficioso/beneficioso<adj><m><sg>:8 *cooperar/*cooperar:9 mediante/mediante<pr>:10 
el/el<det><def><m><sg>:11 establecimiento/establecimiento<n><m><sg>:12 de/de<pr>:0 un/uno<det><ind><m><sg>:13 programa/programa<n><m><sg>:14
común/común<adj><mf><sg>:15 de/de<pr>:16 investigaciones/investigación<n><f><pl>:17 y/y<cnjcoo>:18 de/de<pr>:19 desarollo/desarrollo<n><m><sg>:20
;<sent>:0</s>

Running ReTraTos_lex[edit]

You will need the header and footer of a bilingual dictionary in two separate files, for example, dic_header.txt and dic_footer.txt (see the examples in the package). Then the program for generating the dictionary (ReTraTos_lex) can be called like this: 

$ ReTraTos_lex -s ./es/es_it.aligned.A3.final.al -t ./it/es_it.aligned.A3.final.al -b dic_header.txt -e dic_footer.txt 

PRE-PROCESSAMENTO

        Reading the examples ...  100000 examples read
        Reading the examples ...  100000 examples read

GERANDO LEXICO

        Generating source-target dictionary ... OK
        Generating target-source dictionary ... OK
        Processing bilingual dictionary ... OK
        Generalising bilingual dictionary ... OK
        Cleaning equal attributes ... OK

IMPRIMINDO LEXICO

        Printing bilingual dictionary ... OK

The output file will be the .dix file.

Troubleshooting[edit]

Wrongly tagged

If you are getting many messages like this, or even just one, it means that there is an error in one of your alignment files. 

WARNING: (Entrada::le_sentenca): String O<cnjcoo>.<sent>V<num><mf><pl> is wrongly tagged
WARNING: (Entrada::le_sentenca): String )<rpar>.<sent> is wrongly tagged
WARNING: (Entrada::le_sentenca): String a<pr>)<rpar> is wrongly tagged
WARNING: (Entrada::le_sentenca): String 1991<num>.<sent> is wrongly tagged

...

The alignments in the files must be in the format of lemma<tags>:<alignment>, e.g. 1991<num>:1 or pero<cnjcoo>:2. Forbidden are the alignments which have random strings between tags, e.g. O<cnjcoo>.<sent>V<num><mf><pl> is a set of three lemma/tag pairs with the same alignment. Probably the punctuation has not been separated out correctly from the text with spaces.

It is possible to fix this manually, by going in and either removing punctuation, or inserting spaces and null-alignments, but it might be better if there are many warning messages to re-process the corpus.

See also[edit]

External links[edit]

Further reading[edit]

Retrieved from "https://wiki.apertium.org/w/index.php?title=ReTraTos&oldid=75658"