User:Skh/Application GSoC 2010 - Revision history

Skh: /* Implementation */

2010-04-08T19:57:58Z

Implementation

Skh: /* Additional types of multiwords */

2010-04-08T19:49:36Z

Additional types of multiwords

Skh: /* Ambiguity */

2010-04-08T19:49:09Z

Ambiguity

Skh: /* Discontiguous multiwords (verb ... particle) */

2010-04-08T19:47:57Z

Discontiguous multiwords (verb ... particle)

Skh: /* Timeline */

2010-04-08T19:06:21Z

Timeline

Skh: /* Formalisms */

2010-04-08T18:56:44Z

Formalisms

Skh: /* Complex multiwords (adj-noun) */

2010-04-08T18:40:22Z

Complex multiwords (adj-noun)

Skh: /* Supported multiword constructs */

2010-04-08T15:41:20Z

Supported multiword constructs

Skh: /* Why is it you are interested in machine translation? */

2010-04-08T13:03:32Z

Why is it you are interested in machine translation?

Skh: /* Additional types of multiwords */

2010-04-08T13:03:10Z

Additional types of multiwords

@@ Line 164: / Line 164: @@
 Discontiguous multiwords can be described by regular expressions, so it should be possible to describe and handle them with the existing FST libraries used in apertium.
-Conceptually, the multiword module could fit well into <tt>lttoolbox</tt>.
+Conceptually, the multiword module could fit well into <tt>lttoolbox</tt>, and share code for handling dictionaries and processing the data stream.
 ==== Integration into the apertium pipeline ====

@@ Line 151: / Line 151: @@
 * cases where a particle verb in some cases needs to be reordered and the particle comes first (because the finite verb needs to be in sentence-second position)
-* separable verbs
 * multiwords that are complex and discontiguous at the same time

@@ Line 134: / Line 134: @@
 ==== Ambiguity ====
-It is possible to construct many ambiguous cases with multiwords, especially with those of the discontiguous type.
+It is also possible to construct many ambiguous cases with discontiguous multiwords.
 As an example, consider the following sentences:

@@ Line 111: / Line 111: @@
 ==== Discontiguous multiwords (verb ... particle) ====
-These multiwords consist of a verb and a particle, which do not stand next to each other in the sentence. What can stand between the verb and the particle depends on the language, some are more strict than others. For some, phrase-based rules are necessary, others have position-based rules.
+These multiwords consist of a verb and a particle, which do not stand next to each other in the sentence. These constructs vary wildly across languages, so flexible ways to define them in the dictionary are necessary.
+An example is the english phrasal verb ''to throw sth. away''. With increasing difficulty, sentence structures like the following need to be recognized and generated:
-Example:
-* to make something up
+* ''I throw away the letter.'': This can already be handled, it is the existing type of multiword with inner inflection.
-''dictionary example TBD''
+* ''I throw it away.'' has a pronoun between verb and particle,
+* ''I throw the letter away.'' has a noun phrase, and
+* ''I throw the big red nasty letter from my brother away.'' still a noun phrase, but a far more elaborate one.
+To describe the possible patterns of this example in the dictionary, it needs to be defined what can come between the verb and the particle (in this case, the direct object of the verb, whatever form it takes).
-''stream example TBD''
+Another example are separable verbs, which are written as one word in their non-finite forms, but separate in their finite ones, like german ''ankommen'' ("to arrive"):
+* ''Ich komme an.'' ("I arrive.")
+* ''Ich komme am Bahnhof an.'' ("I arrive at the station.")
+* ''Ich komme morgen abend am Bahnhof an.'' ("I arrive at the station tomorrow evening.")
+* ''Ich komme, wenn nichts dazwischen kommt, morgen abend am Bahnhof an.'' ("If nothing unexpected happens, I arrive at the station tomorrow evening.")
+The difference to the example above is that the particle always stands at the end of a clause, even if whole subordinate clauses are inserted, and that it is less important what kinds of phrases stand between verb and particle.
+As discontiguous multiwords are much more varied than the type of complex multiwords described above, I would like to restrict their treatment in this project to phrasal like ''to throw away'' in simple sentences with either a pronoun or a simple noun phrase between the verb and the particle. After the project it can be evaluated if this approach is usable and extendable to more complicated cases or not.
 ==== Ambiguity ====

@@ Line 174: / Line 174: @@
 * '''Deliverable #1: working binary that can parse the multiword dictionary and read in a disambiguated data stream'''
-* Week 5 and 6: recognize and generaten complex multiwords of type adj-noun
+* Week 5 and 6: recognize and generate complex multiwords of type adj-noun
 * Week 7 and 8: recognize discontiguous multiword in unambiguous simple sentences

@@ Line 149: / Line 149: @@
 It is desired to extend the multiword module in the future so that these and other additional types of multiwords can be handled. They are beyond the main scope of this project, though. Some of them might be included if there's time left at the end.
-==== Formalisms ====
+==== Implementation ====
-Formally, a finite state transducer can't handle either complex multiwords as defined above (explicitly listing all possible inflected forms works just fine of course), because it needs to match tags and copy them to the right side, or discontiguous multiwords, because it needs to keep track of what stands between the verb and its particle. Constructs like these can't be part of a regular language, and therefore can't be matched by finite automata.
+Formally, a finite state transducer can't handle complex multiwords as defined above, because to find out whether the two or more analysed word forms agree in the specified categories, it would need to keep track of the first match, and back-reference it in the second. This is outside regular languages (cf. [http://en.wikipedia.org/wiki/Regular_expression#Patterns_for_non-regular_languages Wikipedia on regular expressions]). Explicitly listing all possible inflected forms works just fine of course, and if this is not desired in the dictionary definition itself, multiword definitions can be expanded automatically.
+Discontiguous multiwords can be described by regular expressions, so it should be possible to describe and handle them with the existing FST libraries used in apertium.
-In the field of regular expressions, however, patterns like these are typically handled by extensions to the "regular" expression engine (which then isn't regular any more of course). I therefore plan to research existing regular expression engines (sed, perl/pcre) first to find out how they have solved the problem. I would like to avoid reinventing the wheel, and find a solution that is robust and efficient.
+Conceptually, the multiword module could fit well into <tt>lttoolbox</tt>.
 ==== Integration into the apertium pipeline ====

@@ Line 79: / Line 79: @@
 These multiwords consist of two inflected words, typically an adjective and a noun, which agree with each other in gender, number and case. They are formally not distinguishable from any other adjective-noun pair, but their meaning can't be inferred from the constituent words, and they therefor need a separate entry in the dictionary.
-Currently, multiwords of this type can be handled in the monolingual dictionaries by explicitly defining the adj-noun combination in all its conjugated forms. This works fine in languages with little inflection (see the example for ''dirección general'' on the [[Multiwords]] wiki page), but gets increasingly ugly when a language inflects with more variation, like the slavic or some germanic languages.
+Currently, multiwords of this type can be handled in the monolingual dictionaries by explicitly defining the adj-noun combination in all its conjugated forms. This works fine in languages with little inflection (see the example for ''dirección general'' on the [[Multiwords]] wiki page), but gets increasingly complicated when a language inflects with more variation, like the slavic or some germanic languages.
 More examples:

@@ Line 69: / Line 69: @@
 * '''Compound multiwords''': two words concatenated into one for phonetic or orthographic reasons (Examples: spanish ''del'' < ''de el'', english ''isn't'' < ''is not''
-* '''Multiwords with inner inflection''': groups of two or more words where one is inflected and the others unchanged (Example: english ''record player'')
+* '''Multiwords with inner inflection''': groups of two or more words where one is inflected and the others unchanged (Example: english ''record player'' with plural ''record players'', ''take away'' with past tense ''took away'')
-{{comment|record player → record players :) A better example might be took away < take away - [[User:Francis Tyers|Francis Tyers]] 10:23, 8 April 2010 (UTC)}}
 === Missing multiword constructs ===

@@ Line 46: / Line 46: @@
 == Motivation ==
 === Why is it you are interested in machine translation? ===
-I have been interested in languages for a long time, and I've been already working as a programmer, so my decision to study computational linguistics was a logical conclusion. Machine translation appeals to me because to do it successfully, both current research and real-world engineering methods and consideration of efficiency are necessary.
+I have been interested in languages for a long time, and I've been already working as a software developer, so my decision to study computational linguistics was a logical conclusion. Machine translation appeals to me because to do it successfully, both current research and real-world engineering methods and consideration of efficiency are necessary.
 Also, while there are already translation tools with work well for specific subject areas and languages, there is still pioneer work to be done, especially for languages that don't have that many speakers.

@@ Line 141: / Line 141: @@
 ==== Additional types of multiwords ====
-* cases where a particle verb in some cases needs to be reorded and the particle comes first (because the finite verb needs to be in sentence-second position)
+* cases where a particle verb in some cases needs to be reordered and the particle comes first (because the finite verb needs to be in sentence-second position)
 * separable verbs