Ideas for Google Summer of Code/Eliminate trimming

Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the translation model (FST compiled of bidix) knows of. The practical effect of this is that words missing from bidix are treated the same as words missing from monodix, making debugging harder. This is workaroudable by compiling trimmed and untrimmed FSAs separately for debug and development process and adding more modes and trying to remember which modes go with which but it's error-prone and unmanageable. Furthermore, throwing good information away early is not a good thing, even when bidix is missing some stuff other parts of the pipeline may benefit from the stuff that got thrown out. Ideally we would want to keep maximal amount of stuff intact and usable and only programmatically select what is displayed when: source language or target language, word-form or lemma...

Coding challenge

• Check out different trimming methods in apertium pairs, including HFST trimming and lttoolbox trimming (Norwegian, North Sámi...)
• Figure out and explain trimming in https://github.com/apertium/apertium-sme-nob/ (hint: Makefile.am)

Further reading

• Why we trim
• Unhammer et al. (20xx) Trimming...