Reordering superblanks
Currently there is a major problem with how formatting / superblanks interacts with word/chunk reordering in Apertium.
If the input is
<a id="foobar" href="http://example.com">Foo <b>bar</b>.</a>
and we want to reorder the words, we currently only reorder the words, and don't touch (or even look at) the blanks, since we don't want to mess up the html, so the output becomes
<a id="foobar" href="http://example.com">Бар <b>фоо</b>.</a>
but now the bold has shifted from source word "bar" to the target word that was "foo" in the input.
Ideally, the output should be
<a id="foobar" href="http://example.com"><b>Бар</b> фоо.</a>
Problems
All language pairs do this kind of thing:
$ echo '<i>Perro</i> <b>blanco</b>' |apertium es-en -f html <i>White</i> <b>dog</b>
And those that don't, will at some point mess up whatever formatting they're given.
The problem is not only that we bold or italicise the wrong word, but also that it limits any possibility of accurately finding out which words were reordered during translation. This kind of reordering information would be useful for systems like Mediawiki's Content Translation (see discussion).
A more serious problem, noted by User:Mlforcada and galaxyfeeder in this discussion, is that tags that are in a valid order in t1x can still be moved around inside chunks in t2x, e.g.
input: <i>foo<i> <b>bar fum</b> fie
after t1x: [<i>]^SN{^foo<adj>$[<i> <b>]^bar<n>$}$ ^SV{^fum<adv>$[</b> ]^fie<vblex>$}$
after t2x: [<i>]^SV{^fum<adv>$[</b> ]^fie<vblex>$}$ ^SN{^foo<adj>$[<i> <b>]^bar<n>$}$
The t2x rules may have completely "correct" blank handling in that they output all input superblanks in the correct order, but they have no way of looking at the blanks that are inside the chunks, so they reorder them wrongly.
Possible solution
User:Tino Didriksen's post at http://comments.gmane.org/gmane.comp.nlp.apertium/3921 outlines a solution:
For each format, we need a list of inline/wordbound tags; for HTML this would include <b>, <i> and so on.
Other tags, like <p> are treated similarly to before, but inline tags stick with their words:
Given input string "<p><b><i>My sister lives</i> <u>in Wales</u></b></p>" you turn that into
My <b><i>
sister <b><i>
lives <b><i>
in <b><u>
Wales <b><u>
Now on outputting, we can just put the inline tags on each word – this might mean some tags are unnecessarily duplicated, but that should be fine.
What we need to support something like this in Apertium:
- Each deformatter needs a list of which tags need the inline treatment
- Deformatters have to turn
<p><b><i>foo</i> bar</b></p>into something like[<p>][{<b><i>}]foo [{<b>}]bar[</p>]- As it is, {} is escaped in regular superblanks, so an unescaped {} inside [] would have this special inline-blank meaning.
- Also, reformatters need to close the tags again, turning
[<p>][{<b><i>}]foo [{<b>}]bar[</p>]into<p><b><i>foo</i></b> <b>bar</b></p>
- Pretransfer will have to distribute the tags when splitting, so
[{<i>}]^foo<vblex>+bar<prn># fie$turns into[{<i>}]^foo# fie<vblex>$ [{<i>}]^bar<prn>$ - Transfer modules have to treat the inline-blanks differently from other superblanks
- All regular superblanks are output before the rule-output, ensuring they're not reordered or deleted
- Regular unmarked/"free" blanks (spaces, etc) are output whenever there's a
<b/>in the rule; each such blank is output at most once, and if they're not used up by the rule, they're output after the rule output. Thus we output all and only those unanalysed chars that were in the input, and in the same order. - Inline formatting blanks are output before each
<lu/>. We look at the clips, and if e.g. the rule is about to output<clip pos="1" part="lemh"/>, we use the inline-blank (if any) from the first word- Note that if a word is deleted, we should be fine; removing an inline blank will not mess up HTML etc.
- To deal with the t2x/chunk-reordering issue mentioned above,
<b pos="N"/>only outputs "free" blanks (unanalysed chars like spaces) or a plain space if we've used up all the freeblanks (the pos="N" no longer has any significance and is ignored). Any non-inline superblanks that were between patterns in the input (ie. the stuff that would go in the b element before) should be output before the rule. This would remove the whole issue since now only inline-blanks or unanalysed chars are allowed inside chunks. - This would also deal with the issue mentioned by Sergio that transfer rule writers forget to output b elements, or output them in the wrong order.
See also
- Format handling
- https://www.mediawiki.org/wiki/Content_translation/Markup#Annotation_mapping_using_translation_subsequence_approximation how mediawiki bravely works around Apertium's limitations
