User:Khannatanmai/GSoC2020 Final Report
This is the final report for the GSoC 2020 project initially titled "Modifying the apertium stream format and eliminating dictionary trimming", which ultimately became "Modifying the apertium stream format to introduce wordbound blanks and solve the markup reordering problem".
Format/Markup handling has been a problem in Apertium and is described in detail here. The way format handling was done is using superblanks that protect markup information inside superblanks during the translation process. While this works well to protect the formatting of the document, during translation, words/phrases move around, get deleted, split, merge, etc., and the markup information on the words needs to go through these with the words, otherwise we will end up with erroneous markup, which is what happened in Apertium.
$ echo '<i>Perro</i> <b>blanco</b>' | apertium spa-eng -f html <i>White</i> <b>dog</b>
A new kind of blank was proposed - wordbound blank. This blank contains any information that needs to stay attached to a word/phrase during the translation process. At this point, this is markup information, but introducing wordbound blanks basically introduces alignments in Apertium and there are several use cases for information inside wordbound blanks.
Wordbound blanks will be denoted by double square brackets and will always appear right before the Lexical Unit to which they are attached.
If there is no Lexical Unit in the stream (before the morph analyser and after the generator), then we have an end wblank as well.
[[wordboundblank]]word[[/]] word2 word3 [[wordboundblank]]word4[[/]]
A new deformatter and reformatter was written, that converts markup into either superblanks or wordbound blanks, and each module in the entire pipeline was modified to work with wordbound blanks.
When wordbound blanks are on text that isn't tokenised using LUs, there's a closing wordbound blank
to limit it's span. After the input is tokenised by the analyser, the program
apertium-wblank-attach attaches wordbound blanks to all the LUs inside the span of that wordbound blank and removes the closing wordbound blanks from the stream. Right before the generator, the opposite process happens with the
apertium-wblank-detach - closing wordbound blanks are put in the stream again in preparation for the reformatter.
Here's a snippet of the process:
Input: <i>Perro</i> <b>blanco</b> tf-extract: [[t:i:4_tPUA]]Perro[[/]] [[t:b:ls9z2Q]]blanco[[/]] Analyser (lt-proc -z): [[t:i:4_tPUA]]^Perro/Perro<n><m><sg>$[[/]] [[t:b:ls9z2Q]]^blanco/blanco<n><m><sg>/blanco<adj><m><sg>$[[/]] apertium-wblank-attach: [[t:i:4_tPUA]]^Perro/Perro<n><m><sg>$ [[t:b:ls9z2Q]]^blanco/blanco<n><m><sg>/blanco<adj><m><sg>$ . . . Transfer Output: [[t:b:ls9z2Q]]^White<adj><sint>$ [[t:i:4_tPUA]]^dog<n><sg>$ apertium-wblank-detach: [[t:b:ls9z2Q]]^White<adj><sint>$[[/]] [[t:i:4_tPUA]]^dog<n><sg>$[[/]] Generator (lt-proc -g): [[t:b:ls9z2Q]]White[[/]] [[t:i:4_tPUA]]dog[[/]] tf-inject: <b>White</b> <i>dog</i>
Here is the full pipeline:
$ echo "<i>Perro</i> <b>blanco</b>" | tf-extract | lt-proc -z 'spa-eng.automorf.bin'. | apertium-wblank-attach | apertium-tagger -z -g $2 'spa-eng.prob' | apertium-pretransfer -z | lt-proc -z -b 'spa-eng.autobil.bin' | lrx-proc -z -m 'spa-eng.autolex.bin' | apertium-transfer -z -b 'apertium-eng-spa.spa-eng.t1x' 'spa-eng.t1x.bin' | apertium-interchunk -z 'apertium-eng-spa.spa-eng.t2x' 'spa-eng.t2x.bin' | apertium-postchunk -z 'apertium-eng-spa.spa-eng.t3x' 'spa-eng.t3x.bin'. | apertium-wblank-detach | lt-proc -z -g 'spa-eng.autogen.bin' | lt-proc -z -p 'spa-eng.autopgen.bin' | tf-inject <b>White</b> <i>dog</i>