Ideas for Google Summer of Code/Robust tokenisation - Revision history

Unhammer at 12:31, 27 March 2024

2024-03-27T12:31:43Z

Unhammer: /* Coding challenge */

2024-03-25T10:23:56Z

Coding challenge

Unhammer: /* Coding challenge */

2024-03-25T10:23:19Z

Coding challenge

Unhammer at 10:18, 25 March 2024

2024-03-25T10:18:17Z

Unhammer at 10:07, 25 March 2024

2024-03-25T10:07:28Z

Unhammer: /* Task */

2024-03-25T10:04:13Z

Task

Unhammer: /* Task */

2024-03-25T10:03:35Z

Task

Unhammer: /* Further readings */

2024-03-25T10:02:27Z

Unhammer: /* Task */

2024-03-25T10:01:36Z

Task

Unhammer at 07:12, 24 February 2023

2023-02-24T07:12:21Z

@@ Line 9: / Line 9: @@
 ** Can we implement "compound-only-right" – ie. have entries that are *only* allowed to end compounds, not appear as words on their own?
 ** Can we implement more nuanced compound restrictions in lttoolbox without going full flag diacritic, e.g. "ulve+jakt" (with -e- when it's the first word) but "prærie+ulv+jakt" (no -e- when it's a middle part)
+** Can we allow limited reanalysis as compound, e.g. if "Vertriebsprinzip" is in the dix as one lemma but we [https://www.reddit.com/r/LanguageTechnology/comments/1bo7svi/rebuilding_german_compound_words/ still want to know that it is a compound] of vertrieb+prinzip (but where do we stop, what about ver+trieb, false analyses etc.)
 * Support for spaceless ortographies / no-space scripts like Japanese, Thai (perhaps using "plugins", e.g. mecab for Japanese)

← Older revision		Revision as of 10:23, 25 March 2024
Line 46:		Line 46:


	2. Make a valiant effort to solve an open issue from https://github.com/apertium/lttoolbox/issues/ / https://github.com/hfst/hfst/issues/ / https://github.com/apertium/apertium-separable/issues/ . Note: These are challenging, but they are exactly the kind of coding you will be working on for this project.		2. Make a valiant effort to solve an open issue from [https://github.com/apertium/lttoolbox/issues/ lttoolbox], [https://github.com/hfst/hfst/issues/ hfst] or [https://github.com/apertium/apertium-separable/issues/ apertium-separable]. Note: These are challenging (that's why they're open issues), but they are exactly the kind of coding you will be working on for this project.

	== Further readings ==		== Further readings ==

@@ Line 27: / Line 27: @@
 ==Coding challenge==
-Write a program that uses data from Unicode to classify characters in an input stream into alphabetic and non-alphabetic.
+. Write a program that uses data from Unicode to classify characters in an input stream into alphabetic and non-alphabetic.
 e.g.
@@ Line 45: / Line 45: @@
 </pre>
+. Make a valiant effort to solve an open issue from https://github.com/apertium/lttoolbox/issues/ / https://github.com/hfst/hfst/issues/ / https://github.com/apertium/apertium-separable/issues/ . Note: These are challenging, but they are exactly the kind of coding you will be working on for this project.
 == Further readings ==

@@ Line 4: / Line 4: @@
 * Names etc. with accent not in alphabet or dictionary: ''Müller'' should be one token even if ü does not appear in dictionary or alphabet
+* Compounding strategies
-* Compounding strategies: banana-door may be 1 or 2 tokens depending on dictionary writers preferences and should not be effected if - is unicode character MINUS-HYPHEN, HYPHEN or EN-DASH, a strategy must also consider if - is replaced with ZERO-WIDTH JOINER or even NON-BREAKING SPACE
+** banana-door may be 1 or 2 tokens depending on dictionary writers preferences and should not be effected if - is unicode character MINUS-HYPHEN, HYPHEN or EN-DASH, a strategy must also consider if - is replaced with ZERO-WIDTH JOINER or even NON-BREAKING SPACE
-* More compounding: there's a possibility of generalising https://github.com/apertium/lttoolbox/pull/139 to more "compounds-with-spaces"?
+** Can we combine compounds with multiwords / words-with-spaces in general? ( https://github.com/apertium/lttoolbox/pull/139 implemented support for a limited form)
-* No-space scripts
+** Can we implement "compound-only-right" – ie. have entries that are *only* allowed to end compounds, not appear as words on their own?
+** Can we implement more nuanced compound restrictions in lttoolbox without going full flag diacritic, e.g. "ulve+jakt" (with -e- when it's the first word) but "prærie+ulv+jakt" (no -e- when it's a middle part)
+* Support for spaceless ortographies / no-space scripts like Japanese, Thai (perhaps using "plugins", e.g. mecab for Japanese)
 I have also found out that some languages abuse the tokenisation algorithms currently in use by defining characters like hyphens as not word characters effectively making apertium treat them like whitespace... this probably (I'm not sure I fully understand this hack) allows haphazard compounding like banana-aeroplane-combucha-mocca-latte kind-of-stuff but has already been problematic when improving the pipeline e.g. in gsoc 2020. The upgrade path for robust tokenisation has to however consider that users of this hack will still be able to work without regressions...

@@ Line 5: / Line 5: @@
 * Names etc. with accent not in alphabet or dictionary: ''Müller'' should be one token even if ü does not appear in dictionary or alphabet
 * Compounding strategies: banana-door may be 1 or 2 tokens depending on dictionary writers preferences and should not be effected if - is unicode character MINUS-HYPHEN, HYPHEN or EN-DASH, a strategy must also consider if - is replaced with ZERO-WIDTH JOINER or even NON-BREAKING SPACE
-* No-space scripts (is this solved by https://github.com/chanlon1/tokenisation ?)
+* More compounding: there's a possibility of generalising https://github.com/apertium/lttoolbox/pull/139 to more "compounds-with-spaces"?
+* No-space scripts
 I have also found out that some languages abuse the tokenisation algorithms currently in use by defining characters like hyphens as not word characters effectively making apertium treat them like whitespace... this probably (I'm not sure I fully understand this hack) allows haphazard compounding like banana-aeroplane-combucha-mocca-latte kind-of-stuff but has already been problematic when improving the pipeline e.g. in gsoc 2020. The upgrade path for robust tokenisation has to however consider that users of this hack will still be able to work without regressions...
@@ Line 14: / Line 15: @@
 ** More or less fixed in https://github.com/apertium/lttoolbox/issues/81 ?
 ** Is this still unfixed in HFST?
+* Support spaceless ortographies
+** See https://github.com/chanlon1/tokenisation
+** and [[Tokenisation for spaceless orthographies]]
 * Allow dictionary developers more control over tokenisation

@@ Line 11: / Line 11: @@
 ==Task==
-* Update [[lttoolbox]] to be fully Unicode compliant with regards to alphabetical symbols.
+* Update [[lttoolbox]] to be fully Unicode compliant with regards to alphabetical symbols, we want "*Müller" and not "*Mu *ü *ller" regardless of the alphabet in .dix
 ** More or less fixed in https://github.com/apertium/lttoolbox/issues/81 ?
 * Allow dictionary developers more control over tokenisation

@@ Line 42: / Line 42: @@
 * https://github.com/hfst/hfst/blob/master/tools/src/hfst-tokenize.cc
 * https://unicode.org/reports/tr29/
+* [[Tokenisation_for_spaceless_orthographies]]
 [[Category:Ideas_for_Google Summer of Code|Robust tokenisation]]

← Older revision		Revision as of 07:12, 24 February 2023
Line 44:		Line 44:

	[[Category:Ideas_for_Google Summer of Code\|Robust tokenisation]]		[[Category:Ideas_for_Google Summer of Code\|Robust tokenisation]]
			[[Category:Tokenisation]]