Ideas for Google Summer of Code/Robust tokenisation

Task

• Update lttoolbox to be fully Unicode compliant with regards to alphabetical symbols.

Coding challenge

• Remove all multiwords from an Apertium language pair and put them in an apertium-separable dictionary.
• Make sure that the output before/after is identical.

Further readings

• https://github.com/hfst/hfst/blob/master/tools/src/hfst-tokenize.cc
• https://unicode.org/reports/tr29/