Reordering superblanks - Revision history

Tino Didriksen: /* Possible solution */

2020-06-21T18:32:58Z

Possible solution

Tino Didriksen: Fix nowiki tag

2020-05-26T15:22:59Z

Fix nowiki tag

Unhammer: /* Problems */

2017-05-30T09:33:14Z

Problems

Unhammer: /* Closing inline-blanks */

2017-03-20T10:30:37Z

Closing inline-blanks

Unhammer: /* Possible solution */

2017-03-20T10:24:13Z

Possible solution

Unhammer: /* See also */

2017-03-17T13:27:37Z

Junzay: /* Implementation(s) */

2016-08-03T14:09:17Z

Implementation(s)

Unhammer: /* See also */

2016-06-28T20:00:26Z

Unhammer: /* Possible solution */

2016-05-25T08:38:14Z

Possible solution

Unhammer: /* Possible solution */

2016-05-25T08:29:06Z

Possible solution

@@ Line 39: / Line 39: @@
 ==Possible solution==
-[[User:Tino Didriksen]]'s post at http://comments.gmane.org/gmane.comp.nlp.apertium/3921 outlines a solution (a closely-related solution, with some additional details, is described in [[User:Mlforcada]]'s entry on a [[Ideas_for_Google_Summer_of_Code/superblank_handling_algorithm]]).
+[[User:Tino Didriksen]]'s post at https://www.mail-archive.com/apertium-stuff@lists.sourceforge.net/msg04449.html outlines a solution (a closely-related solution, with some additional details, is described in [[User:Mlforcada]]'s entry on a [[Ideas_for_Google_Summer_of_Code/superblank_handling_algorithm]]).
 For each format, we need a list of '''inline/wordbound tags'''; for HTML this would include &lt;b&gt;, &lt;i&gt; and so on.

@@ Line 68: / Line 68: @@
 #* All '''regular superblanks''' are output before the rule-output
 #** This means they cannot be reordered or deleted, solving the t2x/chunk-reordering issue mentioned above.  This also deals with the [https://sourceforge.net/p/apertium/mailman/apertium-stuff/thread/20cf28cd0904300204v45f35e51i118f4d146f83748@mail.gmail.com/ issue mentioned by Sergio] that transfer rule writers forget to output b elements, or output them in the wrong order.
-#* Regular unmarked blanks, '''freeblanks''' (spaces, etc that are not inside <code><nowiki>[]</code></nowiki>) which are immediately before a word are output whenever there's a <code><nowiki><b/></nowiki></code> in the rule; each such blank is output exactly once. If they're not used up by b-elements in the rule, the remaining freeblanks are output after the rule output. Thus we output all and only those unanalysed chars that were in the input, and in the same order.
+#* Regular unmarked blanks, '''freeblanks''' (spaces, etc that are not inside <code><nowiki>[]</nowiki></code>) which are immediately before a word are output whenever there's a <code><nowiki><b/></nowiki></code> in the rule; each such blank is output exactly once. If they're not used up by b-elements in the rule, the remaining freeblanks are output after the rule output. Thus we output all and only those unanalysed chars that were in the input, and in the same order.
 #** The pos="N" in <code><nowiki><b pos="N"/></nowiki></code> no longer has any significance and is ignored.
 #* Inline '''format blanks''' are output before each <code><nowiki><lu/></nowiki></code>. Since they may be reordered, split or duplicated in the output, we look at the clips to find out which formatblank goes before which lu. So if e.g. the rule is about to output <code><nowiki><clip pos="1" part="lemh"/></nowiki></code>, we might use the inline-blank (if any) from the first word, We use a prioritised list, so prefer part="lemh" over part="tags", and the rule writer may override this by giving a position in the lu-attribute blankfrom.

@@ Line 34: / Line 34: @@
 </pre>
 The t2x rules may have completely "correct" blank handling in that they output all input superblanks in the correct order, but they have no way of looking at the blanks that are inside the chunks, so they reorder them wrongly.
+This is important to people: <blockquote cite="@tarfandy">People in the real world don't care about a tiny increase in BLEU score; they want tags handled properly! #eamt2017<br/>https://twitter.com/tarfandy/status/869195419494096897</blockquote>
 ==Possible solution==

@@ Line 73: / Line 73: @@
 ===Closing inline-blanks===
-From https://github.com/junaidiiith/Apertium_Code/issues/7 , archived in case that goes away:
 Consider the html input <code><nowiki><i>de jour?</i></nowiki></code>. This may be tokenised by lt-proc as "de jour" and "?" (with the apertium-deshtml|lt-proc we get <code><nowiki>[<i>]^de jour/de jour<adv>$^?/?<sent>$[<\/i>]</nowiki></code>, but our deformatter doesn't (and can't) know where the token borders are.
@@ Line 88: / Line 87: @@
 * <code><nowiki><i>de</i> jour? Yes</nowiki></code> →deform→ <code><nowiki>[{<i>}]de[] jour? Yes</nowiki></code> →translate→ <code><nowiki><i>av</i> dagen? Ja</nowiki></code>
 * <code><nowiki><i>de jour?</i><div>Yes</div></nowiki></code> →deform→ <code><nowiki>[{<i>}]de jour?[<div>]Yes[</div>]</nowiki></code> →translate→ <code><nowiki><i>dagens?</i><div>Ja</div></nowiki></code>
+(Originally discussed at https://github.com/junaidiiith/Apertium_Code/issues/7 )
+However, this should only matter when we're at the "edges" of the pipeline. When a word is tokenised, we know that an inline blank stops having effect on seeing the <code><nowiki>$</nowiki></code>. So e.g. pretransfer and transfer don't have to put a [] after every single token.
+The rule: inline blanks have effect until the next superblank or the <code><nowiki>$</nowiki></code> of a token.
 ==Implementation(s)==

@@ Line 69: / Line 69: @@
 #* Inline '''format blanks''' are output before each <code><nowiki><lu/></nowiki></code>. Since they may be reordered, split or duplicated in the output, we look at the clips to find out which formatblank goes before which lu. So if e.g. the rule is about to output <code><nowiki><clip pos="1" part="lemh"/></nowiki></code>, we might use the inline-blank (if any) from the first word, We use a prioritised list, so prefer part="lemh" over part="tags", and the rule writer may override this by giving a position in the lu-attribute blankfrom.
 #** Note that if a word is deleted, we should be fine; removing an inline blank will not mess up HTML etc.
-# The reformatter also needs to know where to close the tags. We can't just close the tags on whitespace, since we can have inconditionals and multiwords. So we need the end of the pipeline to be something like <code><nowiki>foo [{<b>}]bar[]!</nowiki></code> so we can get <code><nowiki>foo <b>bar</b>!</nowiki></code> instead of <code><nowiki>foo <b>bar!</b></nowiki></code> (or vice versa) – similarly, for the tokenise-as-you-analyse to know how to distribute inline blanks on the correct tokens: <code><nowiki><b>foo</b>?</nowiki></code> vs <code><nowiki><b>foo?</b></nowiki></code> (if lt-proc sees just the opening tag, it doesn't know if the "!" should have a preceding <code><nowiki>[{<b>}]</nowiki></code> or not); see https://github.com/junaidiiith/Apertium_Code/issues/7 for more examples.
+# The reformatter also needs to know where to close the tags. We can't just close the tags on whitespace, since we can have inconditionals and multiwords. So we need the end of the pipeline to be something like <code><nowiki>foo [{<b>}]bar[]!</nowiki></code> so we can get <code><nowiki>foo <b>bar</b>!</nowiki></code> instead of <code><nowiki>foo <b>bar!</b></nowiki></code> (or vice versa) – similarly, for the tokenise-as-you-analyse to know how to distribute inline blanks on the correct tokens: <code><nowiki><b>foo</b>?</nowiki></code> vs <code><nowiki><b>foo?</b></nowiki></code> (if lt-proc sees just the opening tag, it doesn't know if the "!" should have a preceding <code><nowiki>[{<b>}]</nowiki></code> or not); see [[#Closing inline-blanks]] for more examples.
+===Closing inline-blanks===
+From https://github.com/junaidiiith/Apertium_Code/issues/7 , archived in case that goes away:
+Consider the html input <code><nowiki><i>de jour?</i></nowiki></code>. This may be tokenised by lt-proc as "de jour" and "?" (with the apertium-deshtml|lt-proc we get <code><nowiki>[<i>]^de jour/de jour<adv>$^?/?<sent>$[<\/i>]</nowiki></code>, but our deformatter doesn't (and can't) know where the token borders are.
+If we turn this into <code><nowiki>[{<i>}]de [{<i>}]jour?</nowiki></code> then lt-proc will fail to notice the multiword expression (since there's more than just a simple space in between); it'll basically break all formatted multiwords.
+But what's worse is the "?" – if we just "split on spaces" and turn <code><nowiki><i>foo?</i> Bar</nowiki></code> into  <code><nowiki>[{<i>}]foo? Bar</nowiki></code> , how does lt-proc know that it's supposed to output <code><nowiki>[{<i>}]^foo/foo<ij>$[{<i>}]^?/?<sent>$ ^Bar/Bar<n>$</nowiki></code> and not <code><nowiki>[{<i>}]^foo/foo<ij>$^?/?<sent>$ ^Bar/Bar<n>$</nowiki></code>? (How does it know that the <code><nowiki>?</nowiki></code> also is in italics?).
+This is also an issue in generation – if the end of the pipeline is <code><nowiki>foo [{<b>}]bar!</nowiki></code> we don't know if that's <code><nowiki>foo <b>bar</b>!</nowiki></code> or <code><nowiki>foo <b>bar!</b></nowiki></code>.
+The simplest solution is that we treat inline blanks as unclosed until the next <code><nowiki>[</nowiki></code>. So when we reach the end-tag <code><nowiki></i></nowiki></code>, we have to we have to start on a superblank, even if the next character is a non-blank (if it is, we immediately close the superblank). So
+* <code><nowiki><i>de jour?</i> Yes</nowiki></code> →deform→ <code><nowiki>[{<i>}]de jour?[] Yes</nowiki></code> →translate→ <code><nowiki><i>dagens?</i> Ja</nowiki></code>
+* <code><nowiki><i>de jour</i>? Yes</nowiki></code> →deform→ <code><nowiki>[{<i>}]de jour[]? Yes</nowiki></code> →translate→ <code><nowiki><i>dagens</i>? Ja</nowiki></code>
+* <code><nowiki><i>de</i> jour? Yes</nowiki></code> →deform→ <code><nowiki>[{<i>}]de[] jour? Yes</nowiki></code> →translate→ <code><nowiki><i>av</i> dagen? Ja</nowiki></code>
+* <code><nowiki><i>de jour?</i><div>Yes</div></nowiki></code> →deform→ <code><nowiki>[{<i>}]de jour?[<div>]Yes[</div>]</nowiki></code> →translate→ <code><nowiki><i>dagens?</i><div>Ja</div></nowiki></code>
 ==Implementation(s)==

@@ Line 76: / Line 76: @@
 ==See also==
+* [[Ideas_for_Google_Summer_of_Code/Automatic_blank_handling]]
 * [[Format handling]]
 * https://www.mediawiki.org/wiki/Content_translation/Markup#Annotation_mapping_using_translation_subsequence_approximation how mediawiki bravely works around Apertium's limitations

@@ Line 72: / Line 72: @@
 ==Implementation(s)==
-* https://github.com/junaidiiith/Apertium_Code/ GsoC2016 project
+* https://github.com/junaidiiith/apertium/tree/blank-handling GsoC2016 project
 * https://github.com/unhammer/apertium/tree/blank-handling older, unfinished implementation of the changes required in apertium-transfer, with notes at https://github.com/unhammer/apertium/blob/blank-handling/blank_notes.org#consequences-of-this-type-of-blank-handling

@@ Line 57: / Line 57: @@
 # Each deformatter needs a list of which tags need the inline treatment
-# Deformatters have to turn <code><nowiki><p><b><i>foo</i> bar</b></p></nowiki></code> into something like <code><nowiki>[<p>][{<b><i>}]foo [{<b>}]bar[</p>]</nowiki></code>
+# Deformatters have to turn <code><nowiki><p><b><i>foo</i> bar</b></p></nowiki></code> into something like <code><nowiki>[<p>][{<b><i>}]foo[] [{<b>}]bar[</p>]</nowiki></code>
 #* As it is, {} is escaped in regular superblanks, so an unescaped {} inside [] would have this special inline-blank meaning.
+#* To avoid ambiguity with multiwords and inconditionals, an inline-blank is closed by the nearest following (possibly empty) <code><nowiki>[]</nowiki></code>
-#* Also, reformatters need to close the tags again, turning <code><nowiki>[<p>][{<b><i>}]foo [{<b>}]bar[</p>]</nowiki></code> into <code><nowiki><p><b><i>foo</i></b> <b>bar</b></p></nowiki></code>
-# Pretransfer will have to distribute the tags when splitting, so <code><nowiki>[{<i>}]^foo<vblex>+bar<prn># fie$</nowiki></code> turns into <code><nowiki>[{<i>}]^foo# fie<vblex>$ [{<i>}]^bar<prn>$</nowiki></code>
+#* Also, reformatters need to close the tags again, turning <code><nowiki>[<p>][{<b><i>}]foo[] [{<b>}]bar[</p>]</nowiki></code> into <code><nowiki><p><b><i>foo</i></b> <b>bar</b></p></nowiki></code>
+# Pretransfer will have to distribute the tags when splitting, so <code><nowiki>[{<i>}]^foo<vblex>+bar<prn># fie$[]</nowiki></code> turns into <code><nowiki>[{<i>}]^foo# fie<vblex>$ [{<i>}]^bar<prn>$[]</nowiki></code>
 # Transfer modules have to treat the inline-blanks differently from other superblanks
 #* All '''regular superblanks''' are output before the rule-output
@@ Line 68: / Line 69: @@
 #* Inline '''format blanks''' are output before each <code><nowiki><lu/></nowiki></code>. Since they may be reordered, split or duplicated in the output, we look at the clips to find out which formatblank goes before which lu. So if e.g. the rule is about to output <code><nowiki><clip pos="1" part="lemh"/></nowiki></code>, we might use the inline-blank (if any) from the first word, We use a prioritised list, so prefer part="lemh" over part="tags", and the rule writer may override this by giving a position in the lu-attribute blankfrom.
 #** Note that if a word is deleted, we should be fine; removing an inline blank will not mess up HTML etc.
-# The generator needs to output with delimiters for the reformatter to know where to close the tags, e.g. using lt-proc -l or similar. We can't just close the tags on whitespace, since we can have inconditionals and multiwords. So we need the end of the pipeline to be something like <code><nowiki>^foo$ [{<b>}]^bar$^!$</nowiki></code> so we can get <code><nowiki>foo <b>bar</b>!</nowiki></code> instead of <code><nowiki>foo <b>bar!</b></nowiki></code>. Alternatively, output an empty superblank where the end tag should be; that could also solve the analysis problem of <code><nowiki><b>foo</b>?</nowiki></code> vs <code><nowiki><b>foo?</b></nowiki></code> (if lt-proc sees just the opening tag, it doesn't know if the "!" should have a preceding <code><nowiki>[{<b>}]</nowiki></code> or not).
+# The reformatter also needs to know where to close the tags. We can't just close the tags on whitespace, since we can have inconditionals and multiwords. So we need the end of the pipeline to be something like <code><nowiki>foo [{<b>}]bar[]!</nowiki></code> so we can get <code><nowiki>foo <b>bar</b>!</nowiki></code> instead of <code><nowiki>foo <b>bar!</b></nowiki></code> (or vice versa) – similarly, for the tokenise-as-you-analyse to know how to distribute inline blanks on the correct tokens: <code><nowiki><b>foo</b>?</nowiki></code> vs <code><nowiki><b>foo?</b></nowiki></code> (if lt-proc sees just the opening tag, it doesn't know if the "!" should have a preceding <code><nowiki>[{<b>}]</nowiki></code> or not).
 ==See also==