Ideas for Google Summer of Code/Robust tokenisation

Task

• Update lttoolbox to be fully Unicode compliant with regards to alphabetical symbols.

Coding challenge

Write a program that uses data from Unicode to classify characters in an input stream into alphabetic and non-alphabetic.

e.g.

echo "This! Is a tešt тест ** % test." | ./classify-symbols
C T
C h
C i 
C s 
X ! 
X  
C I 
C s

...

Further readings

• https://github.com/hfst/hfst/blob/master/tools/src/hfst-tokenize.cc
• https://unicode.org/reports/tr29/