Ideas for Google Summer of Code/Automatic blank handling

From Apertium
Jump to navigation Jump to search

A superblank is something that we don't want to translate, but keep in the output, often things like formatting tags. Currently there is a major problem with how formatting / superblanks interacts with word/chunk reordering in Apertium. Transfer rules can reorder words, but never look at the tags themselves, and have no general way of allowing the tags themselves to be reordered along with the words, without potentially messing up the resulting code (e.g. putting an end tag before a start tag).

There is a solution: let the format handler treat some tags as "inline" (e.g. <em/>, <b/>) and others as "block-level" (e.g. <div>, <p>). Inline tags always stick with their words. If there are several words covered by one inline tag, they're duplicated to each word, and since they're "glued to the word", a transfer rule can move that word around without worrying about blanks. Block-level tags in the input to a transfer rule are always output before the output that a transfer rule gives; so a <p> will never be moved around by a rule.

This GsoC project involves changing the whole Apertium pipeline (lttoolbox, apertium, apertium-lex-tools, and if you have time also CG-3 and HFST) to do this new automatic blank handling, in particular deformatters and the the transfer module(s). A prototype of this work was implemented in 2016, but there is still much work that remains.

Read Format handling and Apertium stream format for background, then Reordering superblanks for some more explanations of the problem and solution.


  • Make deformatters include a list of inline tags, and disperse these to the words covered by them.
  • Make pretransfer disperse tags when splitting lexical units
  • Make transfer output the non-inline blanks before the rule output
  • Make transfer handle inline-blanks, and ignore <b pos="N">
  • Make reformat turn inline-blanks back into real tags
    • [{<i>}]foo [{<i><b>}]bar should become <i>foo</i> <i><b>bar</b></i>
  • Ensure all other modules are fine with the new format for inline blanks

Coding challenges


  1. Make the HTML format handler apertium-deshtml turn "<i>foo <b>bar</b></i>" into "[{<i>}]foo [{<i><b>}]bar"
    • The current way of creating apertium-deshtml from an xml specification run through xsltproc and flex is not likely to be used – if you don't want to mess with that, we recommend you start a new file and link to something like libgumbo1. This'll make your coding challenge something you can build on in the project itself.
  2. If you've completed 1., make apertium-deshtml *not* wrap tags like <p> or <div> in {} (ie. only for inline tags)


  1. Code cleanup:


  1. Build the first proof-of-concept:
    • Compile git clone -b blank-handling
    • check out apertium-en-es from svn and compile it
    • find some input to en-es that triggers a transfer rule that reorders words
    • Manually change the input to apertium-transfer to have inline formatting around words
    • and check if apertium-transfer keeps inline blanks on words, and phrase blanks outside the chunk. Show the input/output.
  2. Find a bug
    • If you didn't find anything amiss in the previous challenge, try triggering different transfer rules until you do (shouldn't take long)
    • Check out master of and see if git diff 374c46e90d7bd8494300dc364b00eb5a813ece79 --stat contains a fix (most likely possibly in the Modifications directory)
  3. Apply changes to to
    • Check git clone -b blank-handling
    • Apply the diff (between that branch and master) from to
    • Try to make it compile – report things that didn't seem to have a 1-1 correspondence

Frequently asked questions

  • none yet, ask us something! :)

See also