Google Summer of Code/Wrap-up Report 2009 - Revision history

Jimregan: mention mentor/administrator/student involvment in the FreeRBMT workshop

2009-09-12T15:54:49Z

mention mentor/administrator/student involvment in the FreeRBMT workshop

Francis Tyers at 15:40, 12 September 2009

2009-09-12T15:40:06Z

Jimregan: try '''to'''

2009-09-12T15:34:35Z

try '''to'''

Jimregan: we need a project description too; taking the one from the socghop page

2009-09-12T15:30:48Z

we need a project description too; taking the one from the socghop page

Jimregan: /* Multi-engine machine translation */ a cheap laugh

2009-09-12T13:54:04Z

Multi-engine machine translation: a cheap laugh

Francis Tyers: /* A translator for Swedish (sv) to Danish (da) */

2009-09-12T13:02:41Z

A translator for Swedish (sv) to Danish (da)

Francis Tyers: /* A translator for Norwegian Bokmål (nb) and Norwegian Nynorsk (nn) */

2009-09-12T13:02:29Z

A translator for Norwegian Bokmål (nb) and Norwegian Nynorsk (nn)

Francis Tyers: /* Conversion of Anubadok */

2009-09-12T13:02:15Z

Conversion of Anubadok

Francis Tyers: /* Trigram part-of-speech tagging */

2009-09-12T13:02:02Z

Trigram part-of-speech tagging

Jimregan: /* Conversion of Anubadok */ more

2009-09-12T12:54:18Z

Conversion of Anubadok: more

@@ Line 2: / Line 2: @@
 The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try to focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 21 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
-This was our first year in Google Summer of Code and we were very fortunate to receive nine student slots. We filled them with some great students and are pleased to report that out of the nine projects, eight were successful. Along with their end of project reports, students have also been invited to write papers along with their mentors for review in an [http://xixona.dlsi.ua.es/freerbmt09/ academic workshop] on free and open-source rule-based machine translation that we are organising with the mentors' money.
+This was our first year in Google Summer of Code and we were very fortunate to receive nine student slots. We filled them with some great students and are pleased to report that out of the nine projects, eight were successful. Along with their end of project reports, students have also been invited to write papers along with their mentors for review in an [http://xixona.dlsi.ua.es/freerbmt09/ academic workshop] on free and open-source rule-based machine translation that we are organising with the mentors' money -- seven of the nine members of the programme committee were GSoC mentors or administrators with Apertium, and the three of the organisers were mentors (and one student).
 ===A translator for Norwegian Bokmål (nb) and Norwegian Nynorsk (nn)===
@@ Line 38: / Line 38: @@
 The core of the system is the translation router, which controls the flow between user and translation server, based on a variety of factors, including the availability of the language pair, the current load on the server, as well as providing a framework to allow these factors to have different priorities on a per-user basis. When used on Amazon EC2, it also takes into account the cost of each translation request. The project is a complete package; as well as the router, it includes a translation daemon, and convenience scripts to ease the rollout of server instances.
+In addition to his work on his project, Víctor is also serving as an organiser for the FreeRBMT workshop.
 ===Conversion of Anubadok===

@@ Line 1: / Line 1: @@
 {{TOCD}}
-The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try to focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 17 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
+The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try to focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 21 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
 This was our first year in Google Summer of Code and we were very fortunate to receive nine student slots. We filled them with some great students and are pleased to report that out of the nine projects, eight were successful. Along with their end of project reports, students have also been invited to write papers along with their mentors for review in an [http://xixona.dlsi.ua.es/freerbmt09/ academic workshop] on free and open-source rule-based machine translation that we are organising with the mentors' money.

@@ Line 1: / Line 1: @@
 {{TOCD}}
-The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try and focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 17 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
+The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try to focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 17 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
 This was our first year in Google Summer of Code and we were very fortunate to receive nine student slots. We filled them with some great students and are pleased to report that out of the nine projects, eight were successful. Along with their end of project reports, students have also been invited to write papers along with their mentors for review in an [http://xixona.dlsi.ua.es/freerbmt09/ academic workshop] on free and open-source rule-based machine translation that we are organising with the mentors' money.

@@ Line 1: / Line 1: @@
 {{TOCD}}
+The [http://www.apertium.org Apertium Project] is a project which works on open-source machine translation and language technology. We try and focus our efforts on lesser-resourced and marginalised languages, but also work with larger languages. To date, we have released translators for 17 language pairs, covering languages spoken by 1.1bn people, from English (est. 500m speakers) to Aranese (est. 4,000 speakers). A similar number of other language pairs are in development. The Apertium software is licensed under the GPL, but in addition (a rarer situation in the machine translation field) so is the '''data''' for all these language pairs.  This means that the data can be re-used by other language projects (e.g. in developing spelling or grammar checkers, thesauri, etc).
 This was our first year in Google Summer of Code and we were very fortunate to receive nine student slots. We filled them with some great students and are pleased to report that out of the nine projects, eight were successful. Along with their end of project reports, students have also been invited to write papers along with their mentors for review in an [http://xixona.dlsi.ua.es/freerbmt09/ academic workshop] on free and open-source rule-based machine translation that we are organising with the mentors' money.

@@ Line 27: / Line 27: @@
 ===Multi-engine machine translation===
-'''Gabriel Synnaeve''' was mentored by '''Francis Tyers''' to work on a module to improve the quality of machine translation by taking translations from different systems and merging their strengths and discard weaknesses. The two systems focussed on in the initial prototype are [http://www.apertium.org Apertium] (rule-based MT) and [http://www.statmt.org/moses/ Moses] (statistical MT) but it can easily be extended to more. The [http://wiki.apertium.org/wiki/Multi-engine_translation_synthesiser#Multi-engine_pipeline idea] behind the system is that for some languages there is often not one MT system which is better than all others, but some are better at some phrases and some are better at others. Thus if we can combine the output of two or more systems with different strengths/weaknesses we can make better translations.
+'''Gabriel Synnaeve''' was mentored by '''Francis Tyers''' to work on a module to improve the quality of machine translation by taking translations from different systems and merging their strengths and discard weaknesses. The two systems focussed on in the initial prototype are [http://www.apertium.org Apertium] (rule-based MT) and [http://www.statmt.org/moses/ Moses] (statistical MT) but it can easily be extended to more. The [http://wiki.apertium.org/wiki/Multi-engine_translation_synthesiser#Multi-engine_pipeline idea] behind the system is that for some languages there is often not one MT system which is better than all others, but some are better at some phrases and some are better at others. Thus, if we can combine the output of two or more systems with different strengths/weaknesses, we can make better translations.
-Perhaps the most exciting aspect of the MEMT project is its potential for use as a research platform for future work on hybrid machine translation, by allowing the researcher to focus only on the algorithms they wish to implement.
+Perhaps the most exciting aspect of the MEMT project is its potential for use as a research platform for future work on hybrid machine translation, by allowing the researcher to focus only on the algorithms they wish to implement. During the project, Gabriel was joined by Francis in person for a 'mini-hackathon', which, despite something of a farcical start involving requests made on IRC for phone calls across Europe on behalf of two people who were in the same city, lead to a greater degree of functionality and modularisation in the code.
 ===Highly scalable web service architecture for Apertium ===

@@ Line 21: / Line 21: @@
 ===A translator for Swedish (sv) to Danish (da)===
-Another language pair adoption, '''Michael Kristensen''', who had previously done some work on this translator, was mentored by [http://javabog.dk/ Jacob Nordfalk], the author of our English to Esperanto translator. As there are very few free linguistic resources for Swedish and Danish the work was pretty much started from scratch, although we took great advantage of the [http://sv.wiktionary.org Swedish Wiktionary]. The translator is only unidirectional from Swedish to Danish and it has an error rate of around 20%.
+Another language pair adoption, '''Michael Kristensen''', who had previously done some work on this translator, was mentored by '''[http://javabog.dk/ Jacob Nordfalk]''', the author of our English to Esperanto translator. As there are very few free linguistic resources for Swedish and Danish the work was pretty much started from scratch, although we took great advantage of the [http://sv.wiktionary.org Swedish Wiktionary]. The translator is only unidirectional from Swedish to Danish and it has an error rate of around 20%.
 The completion of this translator is something of a triumph for Apertium. Begun back in 2005, the project had been neglected for many years. This was the first translator for the Apertium platform that focussed on non-Romance languages, although Michael was forced to redo and correct much of the work that had been done.

@@ Line 6: / Line 6: @@
 This project was accepted as part of our "adopt a language pair" idea
 from our ideas page. Some work had already been done on the translator
-but it was a long way from finished. [http://unhammer.wordpress.com/ Kevin Unhammer] from the
+but it was a long way from finished. '''[http://unhammer.wordpress.com/ Kevin Unhammer]''' from the
-University of Bergen was mentored by [http://www.hum.uit.no/a/trond/ Trond Trosterud]
+University of Bergen was mentored by '''[http://www.hum.uit.no/a/trond/ Trond Trosterud]'''
 from the University of Tromsø. The final result, after an epic effort,
 is a working translator (indeed the first free software translator for nb-nn) that

@@ Line 39: / Line 39: @@
 ===Conversion of Anubadok===
-'''Abu Zaher''' was mentored by '''Kevin Donnelly''' and [http://xixona.dlsi.ua.es/~fran/ Francis Tyers] to convert [http://anubadok.sourceforge.net/ Anubadok], an open-source MT system for English to [http://en.wikipedia.org/wiki/Bengali_language Bengali] to work with the Apertium engine. This was an ambitious project and not all of the goals were realised, but we were able to make the first wide-coverage [http://en.wikipedia.org/wiki/Morphology_(linguistics) morphological analyser / generator] for Bengali and a substantial amount of lexical transfer, so the project was a great success.
+'''Abu Zaher''' was mentored by '''Kevin Donnelly''' and '''[http://xixona.dlsi.ua.es/~fran/ Francis Tyers]''' to convert [http://anubadok.sourceforge.net/ Anubadok], an open-source MT system for English to [http://en.wikipedia.org/wiki/Bengali_language Bengali] to work with the Apertium engine. This was an ambitious project and not all of the goals were realised, but we were able to make the first wide-coverage [http://en.wikipedia.org/wiki/Morphology_(linguistics) morphological analyser / generator] for Bengali and a substantial amount of lexical transfer, so the project was a great success.
 Zaher is also looking at improving the [http://ankur.org.bd/wiki/Documentation#Bangla_Spell_Checking_How-to Ankur] spell checker with information from his analyser / generator, so the work done is already being reused; there is also interest in using the data to create a Bengali stemmer, for more efficient searching/indexing of Bengali texts, and a number of tools which were created to model the various aspects of Bengali inflection will certainly prove useful in other areas of NLP for Bengali.

@@ Line 51: / Line 51: @@
 ===Trigram part-of-speech tagging===
-Zaid Md. Abdul Wahab Sheikh was mentored by [http://www.dlsi.ua.es/~fsanchez/ Felipe Sánchez Martínez] to improve our [http://en.wikipedia.org/wiki/Part-of-speech_tagging part-of-speech tagging] module to use [http://en.wikipedia.org/wiki/Trigram trigrams] instead of [http://en.wikipedia.org/wiki/Bigram bigrams], as well as implementing changes to the training tools to create data for it.
+'''Zaid Md. Abdul Wahab Sheikh''' was mentored by '''[http://www.dlsi.ua.es/~fsanchez/ Felipe Sánchez Martínez]''' to improve our [http://en.wikipedia.org/wiki/Part-of-speech_tagging part-of-speech tagging] module to use [http://en.wikipedia.org/wiki/Trigram trigrams] instead of [http://en.wikipedia.org/wiki/Bigram bigrams], as well as implementing changes to the training tools to create data for it.
 Apertium was originally designed for closely related languages, but is increasingly growing to meet the challenges of translating between more distant languages. One of the unique aspects of Dr. Sanchez's work on Part-of-Speech tagging is the use of target language information to train the tagger, which allows an accurate tagger to be trained using much less data than usual, provided that it is trained on bilingual text. Zaid's work builds on Dr. Sanchez's work with first-order [http://en.wikipedia.org/wiki/Hidden_Markov_model Hidden Markov Models], extending it to second-order HMMs, similarly to [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]. This enables more accurate translation between more distant languages, using the same methods, so that the rest of the Apertium system can continue to grow.

@@ Line 41: / Line 41: @@
 '''Abu Zaher''' was mentored by '''Kevin Donnelly''' and [http://xixona.dlsi.ua.es/~fran/ Francis Tyers] to convert [http://anubadok.sourceforge.net/ Anubadok], an open-source MT system for English to [http://en.wikipedia.org/wiki/Bengali_language Bengali] to work with the Apertium engine. This was an ambitious project and not all of the goals were realised, but we were able to make the first wide-coverage [http://en.wikipedia.org/wiki/Morphology_(linguistics) morphological analyser / generator] for Bengali and a substantial amount of lexical transfer, so the project was a great success.
-Zaher is also looking at improving the [http://ankur.org.bd/wiki/Documentation#Bangla_Spell_Checking_How-to Ankur] spell checker with information from his analyser / generator, so the work done is already being reused; there is also interest in using the data to create a Bengali stemmer, for more efficient searching/indexing of Bengali texts.
+Zaher is also looking at improving the [http://ankur.org.bd/wiki/Documentation#Bangla_Spell_Checking_How-to Ankur] spell checker with information from his analyser / generator, so the work done is already being reused; there is also interest in using the data to create a Bengali stemmer, for more efficient searching/indexing of Bengali texts, and a number of tools which were created to model the various aspects of Bengali inflection will certainly prove useful in other areas of NLP for Bengali.
 ===Apertium going SOA===