Ideas for Google Summer of Code/Eliminate trimming - Revision history

Ilnar.salimzyan: un typo

2020-04-23T10:26:50Z

un typo

Popcorndude: categorize

2020-03-24T19:51:59Z

categorize

TommiPirinen: /* Further reading */

2020-03-20T16:00:12Z

TommiPirinen: /* Coding challenge */

2020-03-20T15:54:59Z

Coding challenge

TommiPirinen: Created page with "Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the trans..."

2020-03-20T14:55:20Z

Created page with "Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the trans..."

New page

Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the translation model (FST compiled of bidix) knows of. The practical effect of this is that words missing from bidix are treated the same as words missing from monodix, making debugging harder. This is workaroudable by compiling trimmed and untrimmed FSAs separately for debug and development process and adding more modes and trying to remember which modes go with which but it's error-prone and unmanageable. Furthermore, throwing good information away early is not a good thing, even when bidix is missing some stuff other parts of the pipeline may benefit from the stuff that got thrown out. Ideally we would want to keep maximal amount of stuff intact and usable and only programmatically select what is displayed when: source language or target language, word-form or lemma...

== Task ==

== Coding challenge ==

* Check out different trimming methods in apertium pairs, including HFST trimming and lttoolbox trimming (Norwegian, North Sámi...)

== Further reading ==

* Unhammer et al. (20xx) Trimming...

← Older revision		Revision as of 10:26, 23 April 2020
Line 1:		Line 1:
	Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the translation model (FST compiled of bidix) knows of. The practical effect of this is that words missing from bidix are treated the same as words missing from monodix, making debugging harder. This is ~~workaroudable~~ by compiling trimmed and untrimmed FSAs separately for debug and development process and adding more modes and trying to remember which modes go with which but it's error-prone and unmanageable. Furthermore, throwing good information away early is not a good thing, even when bidix is missing some stuff other parts of the pipeline may benefit from the stuff that got thrown out. Ideally we would want to keep maximal amount of stuff intact and usable and only programmatically select what is displayed when: source language or target language, word-form or lemma...		Dictionary trimming is a thing in apertium where we remove stuff from monolingual language models (FSTs compiled from monodixes) so they only contain word-forms that the translation model (FST compiled of bidix) knows of. The practical effect of this is that words missing from bidix are treated the same as words missing from monodix, making debugging harder. This is workaroundable by compiling trimmed and untrimmed FSAs separately for debug and development process and adding more modes and trying to remember which modes go with which but it's error-prone and unmanageable. Furthermore, throwing good information away early is not a good thing, even when bidix is missing some stuff other parts of the pipeline may benefit from the stuff that got thrown out. Ideally we would want to keep maximal amount of stuff intact and usable and only programmatically select what is displayed when: source language or target language, word-form or lemma...

@@ Line 14: / Line 14: @@
 * [[Why we trim]]
 * Unhammer et al. (20xx) Trimming...
+[[Category:Ideas_for_Google_Summer_of_Code]]

@@ Line 8: / Line 8: @@
 * Check out different trimming methods in apertium pairs, including HFST trimming and lttoolbox trimming (Norwegian, North Sámi...)
+* Figure out and explain trimming in https://github.com/apertium/apertium-sme-nob/ (hint: Makefile.am)
 == Further reading ==