Difference between revisions of "Hectoralos/GSOC 2019 proposal: Catalan-Italian and Catalan-Portuguese"

From Apertium
Jump to navigation Jump to search
Line 66: Line 66:
 
*coverage: 87.6% (calculated using a Wikipedia corpus with 2.6 million words)
 
*coverage: 87.6% (calculated using a Wikipedia corpus with 2.6 million words)
   
== Google translation ==
+
=== Google translation ===
 
Google and Yandex propose the three languages in question, so one can translate between them using these services. I have analyzed the results of the translation into Catalan from Google (my experience tells me that Yandex gives worse results for Catalan). The numerical results can be seen in the previous sections (ita-cat: 14.0% WER, por-cat: 21.6% WER).
 
Google and Yandex propose the three languages in question, so one can translate between them using these services. I have analyzed the results of the translation into Catalan from Google (my experience tells me that Yandex gives worse results for Catalan). The numerical results can be seen in the previous sections (ita-cat: 14.0% WER, por-cat: 21.6% WER).
   
The results are far from wonderful, especially for the Portuguese-Catalan couple. Google translates using both Spanish and English as bridge languages, as can be seen, for example, by words that appear in these two languages in the final text (supposedly in Catalan) and that were not in the original Italian or Portuguese.
+
The results are far from wonderful, especially for the Portuguese-Catalan couple. Seemingly Google translates using both Spanish and English as bridge languages, as can be seen, for example, by words that appear in these two languages in the final text (supposedly in Catalan) and that were not in the original Italian or Portuguese text. The use of English as intermediate between Romanic languages causes problems known to all users, such as the translation of verb forms p2.pl with elided subject to p2.sg, the incorrect choice of past times in the verbs and the disappearance of some pronouns. Here is an example of the last case of the Italian test text (randomly obtained):
   
  +
Original text (bold mine):
<nowiki>
 
  +
<blockquote>
altri invece ne hanno apprezzato la spontaneità, la tenacia e l'affettuosità
+
altri invece '''ne''' hanno apprezzato la spontaneità, la tenacia e l'affettuosità
</nowiki>
 
  +
</blockquote>
  +
  +
Google translation:
  +
<blockquote>
  +
altres han apreciat '''la seva''' espontaneïtat, tenacitat i afecte
  +
</blockquote>
  +
  +
Post-editet translation:
  +
<blockquote>
  +
altres '''n''''han apreciat l'espontaneïtat, tenacitat i afecte
  +
</blockquote>
  +
  +
It should be add that, although Google translations tend to be more phraseological than the translation obtained by rules, they are also much more difficult to post-edit. The reason is that, while the translation by rules often makes evident and even expected errors, the neuronal translation significantly changes the text, reordering parts of the sentence, removing or putting words and modifying expressions. The evaluation of whether the meaning is the same as the original requires a lot more time. This has been quite clear when I have made the post-edition of both the Apertium and Google translations for the Italian and Portuguese texts.
   
 
=== Workplan ===
 
=== Workplan ===

Revision as of 07:30, 24 March 2019

Contact Information

Name: Hèctor Alòs i Font

Location: Shupashkar, Chuvashia, Russia

E-mail: hectoralos@gmail.com

IRC: hector2

GitHub: hectoralos

Telegram: hectoralos

Skype: hectoralos

Why is it you are interested in machine translation?

I’m a sociolinguist. I'm very interested in creating resources for minoritised languages.

Why is it that you are interested in Apertium?

  • Because Apertium is free/open-source software.
  • Because its community is strongly committed to under-resourced and minoritised/marginalised languages.
  • Because there is lot of good work done and being done in it.
  • Because it is not only machine translation, but also free resources than can be used for other purposes: e.g. dictionaries, morphological analysers, spellcheckers, etc.

Which of the published tasks are you interested in? What do you plan to do?

1.2 Bring a released language pair up to state-of-the-art quality: I'd like to improve the pairs Catalan-Italian and Catalan-Portuguese. In the first case, only the side from Italian to Catalan is available. I plan to create the other side.

My proposal

Title

Improving the Catalan-Italian and Catalan-Portuguese language pairs

Major goals

  • Improving existing translators, obtaining a WER below 15% and a Wikipedia coverage above 91%:
    • Italian to Catalan
    • Portuguese to Catalan
    • Catalan to Portuguese
  • Developing a translator from Catalan to Italian with a WER below 15% and a Wikipedia coverage above 91%.

Unlike it happens with French, compared to other Romance languages, there are not big structural (syntax) differences between Catalan, Italian and Portuguese. If we improve the morphological disambiguation, add several thousands of words in the dictionaries, introduce lexical selection rules and create some more transfer rules, a low WER can be reached.

Reasons why Google and Apertium should sponsor it

The pairs Italian-Catalan, Portuguese-Catalan give unsatisfactory results (WER ≈ 30%, coverage below 85% in Wikipedia corpora). Both were published in 2009 and, apparently, no one has worked on them since then (except for the porting from ca-it to cat-ita and pt-ca to por-cat). In addition, the translation from Catalan to Italian is not developed.

Current situation of the language pairs

Next are a few data on the three existing directions, in its current state of development on GitHub. It must be taken into account that the versions on the web are still those of the apertium-ca-it and apertium-pt-ca pairs, while I have made the calculations from apertium-cat-ita and apertium-por-cat.

Italian to Catalan

  • bidix: 9091 pairs (excluding proper names)
  • Coverage: 81.7% (calculated using a Wikipedia corpus with 3.0 M words)
  • Word error rate (WER): 30.0% (calculated using random Wikipedia texts with a total of 933 words)
  • Word error rate (WER) using Google Translator: 14.0% (calculated using the same test text)

Portuguese to Catalan

  • bidix: 7576 pairs (excluding proper names)
  • Coverage: 84.4% (calculated using a Wikipedia corpus with 3.1 M words)
  • Word error rate (WER): 28.4% (calculated using random Wikipedia texts with a total of 1648 words)
  • Word error rate (WER) using Google Translator: 21.6% (calculated using the same test text)

Catalan to Portuguese

  • bidix: 7576 pairs (excluding proper names)
  • coverage: 87.6% (calculated using a Wikipedia corpus with 2.6 million words)

Google translation

Google and Yandex propose the three languages in question, so one can translate between them using these services. I have analyzed the results of the translation into Catalan from Google (my experience tells me that Yandex gives worse results for Catalan). The numerical results can be seen in the previous sections (ita-cat: 14.0% WER, por-cat: 21.6% WER).

The results are far from wonderful, especially for the Portuguese-Catalan couple. Seemingly Google translates using both Spanish and English as bridge languages, as can be seen, for example, by words that appear in these two languages in the final text (supposedly in Catalan) and that were not in the original Italian or Portuguese text. The use of English as intermediate between Romanic languages causes problems known to all users, such as the translation of verb forms p2.pl with elided subject to p2.sg, the incorrect choice of past times in the verbs and the disappearance of some pronouns. Here is an example of the last case of the Italian test text (randomly obtained):

Original text (bold mine):

altri invece ne hanno apprezzato la spontaneità, la tenacia e l'affettuosità

Google translation:

altres han apreciat la seva espontaneïtat, tenacitat i afecte

Post-editet translation:

altres n'han apreciat l'espontaneïtat, tenacitat i afecte

It should be add that, although Google translations tend to be more phraseological than the translation obtained by rules, they are also much more difficult to post-edit. The reason is that, while the translation by rules often makes evident and even expected errors, the neuronal translation significantly changes the text, reordering parts of the sentence, removing or putting words and modifying expressions. The evaluation of whether the meaning is the same as the original requires a lot more time. This has been quite clear when I have made the post-edition of both the Apertium and Google translations for the Italian and Portuguese texts.

Workplan

Week Dates Goals Bidix
(excluding
proper names)
WER Coverage
Post-application period 10 March - 26 May
  • Find language resources (Wiktionary et al.)
  • Build frequency lists for Italian-Catalan
  • Build frequency lists for Portuguese-Catalan
  • Construct pending tests for the 4 directions
~9,000 (cat-ita)
~7,500 (cat-por)
~30% (cat > ita)
~30% (cat > por)
~30% (por > cat)
~88% (cat > ita)
~82% (ita > cat)
~88% (cat > por)
~84% (por > cat)
1 27 May - 2 June
  • ita > cat
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (ita > cat)
~11,000 (cat-ita) ~85.5% (ita > cat)
2 3 June- 9 June
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (ita > cat)
~13,000 (cat-ita) ~87.5% (ita > cat)
3 10 June - 16 June
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (ita > cat)
~14,000 (cat-ita) <20% (ita > cat) ~89% (ita > cat)
4 17 June - 23 June
  • cat > ita
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (cat > ita)
  • Testvoc cat-ita: closed categories
~15,000 (cat-ita) ~90% (cat > ita)
~90% (ita > cat)
5 24 June - 30 June
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (cat > ita)
  • Testvoc cat-ita: vblex

First evaluation (28 June)

~16,000 (cat-ita) ~90.5% (cat > ita)
~90.5% (ita > cat)
6 1 July - 7 July
  • Expand bilingual dictionary cat-ita
  • Transfer and lexical selection rules (cat > ita)
  • Testvoc cat-ita: adj, adv, np
~17,000 (cat-ita) ~91% (cat > ita)
~91% (ita > cat)
7 8 June - 14 July
  • Transfer and lexical selection rules (cat > ita)
  • Testvoc cat-ita: n
  • Write documentation
~18,000 (cat-ita) <15% (cat > ita)
<15% (ita > cat)
~91.5% (cat > ita)
~91.5% (ita > cat)
8 15 July - 21 July
  • por > cat
  • Expand bilingual dictionary cat-por
  • Disambiguation rules (por > cat)
  • Transfer and lexical selection rules (por > cat)
~9,500 (cat-por) ~87% (por > cat)
9 22 July - 28 July
  • Expand bilingual dictionary
  • Disambiguation rules (por > cat)
  • Transfer and lexical selection rules (por > cat)

Second evaluation (26 July)

~11,500 (cat-por) ~89% (por > cat)
10 29 July - 4 August
  • Expand bilingual dictionary
  • Disambiguation rules (por > cat)
  • Transfer and lexical selection rules (por > cat)
~13,000 (cat-por) <20% (por > cat) ~89.5% (por > cat)
11 5 August - 11 August
  • cat > por
  • Expand bilingual dictionary
  • Transfer and lexical selection rules (cat > por)
  • Testvoc cat-por: closed categories, vblex
~14,500 (cat-por) ~90% (por > cat)
~90% (por > cat)
12 12 August - 18 August
  • Expand bilingual dictionary
  • Transfer and lexical selection rules (cat > por)
  • Testvoc cat-por: adj, adv, np
~16,000 (cat-por) ~90.5% (por > cat)
~90.5% (por > cat)
13 18 August - 25 August
  • Expand bilingual dictionary
  • Transfer and lexical selection rules (cat > por)
  • Testvoc cat-por: n

Final evaluation (26 August)

~17,000 (cat-por) <15% (cat > por)
<15% (por > cat)
~91.0% (por > cat)
~91.0% (por > cat)

List your skills and give evidence of your qualifications

I once got a computer engineering (Universitat Politècnica de Catalunya, 1988), but I’ve forgotten almost everything on programming. I also got a BA on linguistics (Universitat de Barcelona, 2008). I’ve been working on Apertium for several years. In 2011 I created the Esperanto-French pair and prepared new releases of the Esperanto-Catalan and Esperanto-Spanish pairs. I’ve also been working on new releases of the French-Catalan pair (2017, 2018 and currently a third one is been packaged). I’ve mentored and was strongly involved in the GSoC projects on Italian-Sardinian (2016), Catalan-Sardinian (2017) and French-Occitan (2018). In all these cases we released new one-way language pairs just after the end of the GSoC.

Catalan is my mother tongue, and I’ve been studying it at the university. I'm a fluent speaker of Spanish and French. I read fluently in Italian and Portuguese, among other Romance languages, but my knowledge of them is mainly passive and linguistic. That’s why I’ll work mainly translating from Italian and Portuguese into Catalan, but my knowledge of Italian is good enough to create a first version of a translator from Catalan to Italian.

List any non-Summer-of-Code plans you have for the Summer

I can guarantee at least 30 hours per week of work during the whole Summer. As I love this kind of work, I'm sure I'll be engaged quite more.

I have to submit the last two short works for two subjects of the master I am studying on June 5 and 11. It is possible that I will travel on vacation with my family one or two weeks in July. In the unlikely event that one week I do not reach 30 hours of work, I would recover these hours (I have had this workload in other years and there have been no problems).