Ideas for Google Summer of Code/Automatic blank handling

From Apertium
Jump to navigation Jump to search

A superblank is something that we don't want to translate, but keep in the output, often things like formatting tags. Currently there is a major problem with how formatting / superblanks interacts with word/chunk reordering in Apertium. Transfer rules can reorder words, but never look at the tags themselves, and have no general way of allowing the tags themselves to be reordered along with the words, without potentially messing up the resulting code (e.g. putting an end tag before a start tag).

There is a solution: let the format handler treat some tags as "inline" (e.g. <em/>, <b/>) and others as "block-level" (e.g. <div>, <p>). Inline tags always stick with their words. If there are several words covered by one inline tag, they're duplicated to each word, and since they're "glued to the word", a transfer rule can move that word around without worrying about blanks. Block-level tags in the input to a transfer rule are always output before the output that a transfer rule gives; so a <p> will never be moved around by a rule.

This GsoC project involves changing the whole Apertium pipeline (lttoolbox, apertium, apertium-lex-tools, and if you have time also CG-3 and HFST) to do this new automatic blank handling, in particular deformatters and the the transfer module(s). A prototype of this work was implemented in 2016, but there is still much work that remains.

Read Format handling and Apertium stream format for background, then Reordering superblanks for some more explanations of the problem and solution.


Tasks

  • Make deformatters include a list of inline tags, and disperse these to the words covered by them.
  • Make pretransfer disperse tags when splitting lexical units
  • Make transfer output the non-inline blanks before the rule output
  • Make transfer handle inline-blanks, and ignore <b pos="N">
  • Make reformat turn inline-blanks back into real tags
    • [{<i>}]foo [{<i><b>}]bar should become <i>foo</i> <i><b>bar</b></i>
  • Ensure all other modules are fine with the new format for inline blanks

Coding challenges

deformatting

  1. Make the HTML format handler apertium-deshtml turn "<i>foo <b>bar</b></i>" into "[{<i>}]foo [{<i><b>}]bar"
    • The current way of creating apertium-deshtml from an xml specification run through xsltproc and flex is not likely to be used – if you don't want to mess with that, we recommend you start a new file apertium_deshtml2.cc and link to something like libgumbo1. This'll make your coding challenge something you can build on in the project itself.
  2. If you've completed 1., make apertium-deshtml *not* wrap tags like <p> or <div> in {} (ie. only for inline tags)

pretransfer

Code cleanup:

  1. fork https://github.com/unhammer/apertium and check out and compile the master branch; then separately do git clone -b blank-handling https://github.com/junaidiiith/apertium; from junaidiiith, copy over the changes to apertium_pretransfer.cc in your fork of unhammer/apertium, along with the pretransfer tests, and send a pull request to https://github.com/unhammer/apertium

transfer

  1. Compile git clone -b blank-handling https://github.com/unhammer/apertium, install apertium-en-es, find some input that triggers a transfer rule that reorders words. Manually change the input to apertium-transfer to have inline formatting around words, and ensure that apertium-transfer keeps inline blanks on words, and phrase blanks outside the chunk. Show the input/output.
  2. Find a bug in the above blank-handling branch, check out master of https://github.com/junaidiiith/apertium and see if git diff 374c46e90d7bd8494300dc364b00eb5a813ece79 --stat contains a fix (most likely possibly in the Modifications directory)
  3. Apply the changes from transfer.cc to interchunk.cc in the blank-handling branch

Frequently asked questions

  • none yet, ask us something! :)

See also

Retrieved from "https://wiki.apertium.org/w/index.php?title=Ideas_for_Google_Summer_of_Code/Automatic_blank_handling&oldid=61801"