Why we trim - Revision history

Unhammer: /* See also */

2020-05-03T16:06:41Z

Unhammer: /* See also */

2020-05-03T11:26:18Z

Bech: Link to French page

2014-10-08T08:34:47Z

Link to French page

Unhammer: /* See also */

2014-02-14T09:31:34Z

Unhammer at 13:13, 16 October 2012

2012-10-16T13:13:40Z

Unhammer at 13:13, 16 October 2012

2012-10-16T13:13:22Z

Unhammer: bah

2012-10-16T13:11:36Z

bah

Unhammer at 13:11, 16 October 2012

2012-10-16T13:11:14Z

Unhammer at 13:10, 16 October 2012

2012-10-16T13:10:30Z

Unhammer at 12:56, 16 October 2012

2012-10-16T12:56:43Z

@@ Line 23: / Line 23: @@
 * [[Testvoc]]
 * [http://wiki.apertium.eu/index.php/Session_7 Session 7: Data consistency and quality] on wiki.apertium.eu
-* http://tinodidriksen.com/pisg/freenode/logs/%23apertium/2020-05-03.log
 [[Category:Quality control]]

@@ Line 23: / Line 23: @@
 * [[Testvoc]]
 * [http://wiki.apertium.eu/index.php/Session_7 Session 7: Data consistency and quality] on wiki.apertium.eu
+* http://tinodidriksen.com/pisg/freenode/logs/%23apertium/2020-05-03.log
 [[Category:Quality control]]

← Older revision		Revision as of 08:34, 8 October 2014
Line 1:		Line 1:
			[[Pourquoi nous tronquons\|En français]]

	In Apertium language pairs, we keep the monolingual and bilingual dictionaries ''trimmed'', so that all entries from the analyser will have some match in the bidix, and all output from transfer will have some entry in the generator.<ref>Typically this goes for both translation direction, although a language pair only released for one direction might only be trimmed in that direction.</ref>		In Apertium language pairs, we keep the monolingual and bilingual dictionaries ''trimmed'', so that all entries from the analyser will have some match in the bidix, and all output from transfer will have some entry in the generator.<ref>Typically this goes for both translation direction, although a language pair only released for one direction might only be trimmed in that direction.</ref>

@@ Line 18: / Line 18: @@
 ==See also==
 * [[Automatically trimming a monodix]]
+* [[lt-trim]]
 * [[Testvoc]]
 * [http://wiki.apertium.eu/index.php/Session_7 Session 7: Data consistency and quality] on wiki.apertium.eu

@@ Line 6: / Line 6: @@
 # Transfer rules quite often use target language information from bidix to fill in tags etc. If transfer from English to Spanish reads a chunk like "the children", the Spanish determiner needs to get the number and gender information from the ''target language'' noun. It is not enough to look at the output of the source language analyser, number can be changed by bidix for certain nouns, and gender is not even present in the source language. The transfer rule expects to have this information; without it, not only will the noun be output as @lemma, but the determiner will not be generated correctly either. This effect gets even worse with bigger chunks.
 #* One might work around this by having exceptions in the transfer rules to e.g. guess number and gender if bidix doesn't give any, but this leads to an enormous increase in transfer complexity – all tags have to be presumed to be unknown, and developer time is wasted on bug-hunting and workarounds instead of improving translation quality.
-# Although there could be a technical solution (<code>lt-proc -o</code>) to carrying over the source word if it's not in the bidix, this leads to problems with compounds and other multiwords that are split into two lexical units before bidix lookup: What do you do when part of a multiword is unknown? For example, if we have ^writes about/write&lt;vblex&gt;+about&lt;pr&gt;$, this is currently split before bidix lookup into two units ^write&lt;vblex&gt;$ ^about&lt;pr&gt;$, without lemmas, and if only one is unknown after bidix lookup, the other will still translate: ^write&lt;vblex&gt;/escribir&lt;vblex&gt;$ ^about&lt;pr&gt;/@about&lt;pr&gt;$. If, on the other hand, we were to keep the surface form around, we would also have keep it as one unit in bidix lookup, such that if parts of the multiword were unknown, all of it would be marked unknown, giving something like ^@writes about/write&lt;vblex&gt;+@about&lt;pr&gt;$.
+# Although there could be a technical solution to carrying over the source word if it's not in the bidix (<code>lt-proc -o</code>), this leads to problems with compounds and other multiwords that are split into two lexical units before bidix lookup: What do you do when part of a multiword is unknown? For example, if we have ^writes about/write&lt;vblex&gt;+about&lt;pr&gt;$, this is currently split before bidix lookup into two units ^write&lt;vblex&gt;$ ^about&lt;pr&gt;$, without lemmas, and if only one is unknown after bidix lookup, the other will still translate: ^write&lt;vblex&gt;/escribir&lt;vblex&gt;$ ^about&lt;pr&gt;/@about&lt;pr&gt;$. If, on the other hand, we were to keep the surface form around, we would also have keep it as one unit in bidix lookup, such that if parts of the multiword were unknown, all of it would be marked unknown, giving something like ^@writes about/write&lt;vblex&gt;+@about&lt;pr&gt;$.
 #* Can't you just distribute the surface form over the two units? ^writes/write&lt;vblex&gt;$ ^about/about&lt;pr&gt;$! While in this constructed example, the split was at a space, it could be anywhere. The surface form gives no general indication of where. We have multiwords that split in the middle of contractions (^au/à&lt;pr&gt;+le&lt;det&gt;&lt;def&gt;&lt;m&gt;&lt;sg&gt;$), or in the middle of compunds (^vasskokaren/vatn&lt;n&gt;+kokar&lt;n&gt;$)

@@ Line 7: / Line 7: @@
 #* One might work around this by having exceptions in the transfer rules to e.g. guess number and gender if bidix doesn't give any, but this leads to an enormous increase in transfer complexity – all tags have to be presumed to be unknown, and developer time is wasted on bug-hunting and workarounds instead of improving translation quality.
 # Although there could be a technical solution to carrying over the source word if it's not in the bidix, this leads to problems with compounds and other multiwords that are split into two lexical units before bidix lookup: What do you do when part of a multiword is unknown? For example, if we have ^writes about/write&lt;vblex&gt;+about&lt;pr&gt;$, this is currently split before bidix lookup into two units ^write&lt;vblex&gt;$ ^about&lt;pr&gt;$, without lemmas, and if only one is unknown after bidix lookup, the other will still translate: ^write&lt;vblex&gt;/escribir&lt;vblex&gt;$ ^about&lt;pr&gt;/@about&lt;pr&gt;$. If, on the other hand, we were to keep the surface form around, we would also have keep it as one unit in bidix lookup, such that if parts of the multiword were unknown, all of it would be marked unknown, giving something like ^@writes about/write&lt;vblex&gt;+@about&lt;pr&gt;$.
-#* Can't you just distribute the surface form over the two units? ^writes/write&lt;vblex&gt;$ ^about/about&lt;pr&gt;$! While in this constructed example, the split was at a space, it could be anywhere. The surface form gives no general indication of where. We have multiwords that split in the middle of contractions (^au/à&lt;pr&gt;+le&lt;det&gt;&lt;def&lt;m&gt;&lt;sg&gt;$), or in the middle of compunds (^vasskokaren/vatn&lt;n&gt;+kokar&lt;n&gt;$)
+#* Can't you just distribute the surface form over the two units? ^writes/write&lt;vblex&gt;$ ^about/about&lt;pr&gt;$! While in this constructed example, the split was at a space, it could be anywhere. The surface form gives no general indication of where. We have multiwords that split in the middle of contractions (^au/à&lt;pr&gt;+le&lt;det&gt;&lt;def&gt;&lt;m&gt;&lt;sg&gt;$), or in the middle of compunds (^vasskokaren/vatn&lt;n&gt;+kokar&lt;n&gt;$)