https://wiki.apertium.org/w/index.php?action=history&feed=atom&title=Rethinking_tokenisation_in_the_pipeline 2026-05-05T09:49:28Z Revision history for this page on the wiki MediaWiki 1.34.1 https://wiki.apertium.org/w/index.php?title=Rethinking_tokenisation_in_the_pipeline&diff=74189&oldid=prev 2023-02-24T07:12:13Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 07:12, 24 February 2023</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 28:</td> <td colspan="2" class="diff-lineno">Line 28:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>We should also be able to deal with languages that don't write spaces.</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>We should also be able to deal with languages that don't write spaces.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>[[Category:Tokenisation]]</div></td> </tr> </table>

Unhammer https://wiki.apertium.org/w/index.php?title=Rethinking_tokenisation_in_the_pipeline&diff=72523&oldid=prev 2020-06-27T11:49:43Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 11:49, 27 June 2020</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 12:</td> <td colspan="2" class="diff-lineno">Line 12:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>* We did not have [[apertium-separable]] or any other way of merging analyses outside of transfer. </div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>* We did not have [[apertium-separable]] or any other way of merging analyses outside of transfer. </div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>* Unicode support was not as widely used</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>* Unicode support was not as widely used</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>== Current issues ==</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>* Splitting tokens that are alphabetic but not in the &lt;tt&gt;&lt;alphabet&gt;&lt;/tt&gt;, e.g. &lt;tt&gt;^Bala$š^evi$ć&lt;/tt&gt;.</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>== Thoughts ==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>== Thoughts ==</div></td> </tr> <tr> <td colspan="2" class="diff-lineno">Line 22:</td> <td colspan="2" class="diff-lineno">Line 26:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>* retokenisation for multiwords</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>* retokenisation for multiwords</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>** (&gt;1 token, 1 word)</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>** (&gt;1 token, 1 word)</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>We should also be able to deal with languages that don't write spaces.</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Rethinking_tokenisation_in_the_pipeline&diff=72522&oldid=prev 2020-06-27T11:39:52Z

<p>Created page with &quot;At the moment tokenisation is done longest-match left-to-right using the morphological analyser and an alphabet defined for breaking up unknown words. This has some advantage...&quot;</p> <p><b>New page</b></p><div>At the moment tokenisation is done longest-match left-to-right using the morphological analyser and an alphabet defined for breaking up unknown words.<br /> <br /> This has some advantages ... <br /> * It means that multiwords can be included in the dictionary and tokenised as a single unit<br /> * It combines two parts of the pipeline that have complementary information, tokenisation and morphological analysis.<br /> <br /> However it has some disadvantages:<br /> * Not having a standard fixed tokenisation scheme makes using other resources harder, and makes it harder for other people to use our resources. <br /> * Ambiguous paths need to be encoded in the lexicon (using &lt;tt&gt;+&lt;/tt&gt;)<br /> <br /> Historically,<br /> * We did not have [[apertium-separable]] or any other way of merging analyses outside of transfer. <br /> * Unicode support was not as widely used<br /> <br /> == Thoughts ==<br /> <br /> There should be various stages to tokenisation,<br /> * separating whitespace from non-whitespace, no more should we have e.g. &lt;tt&gt;^Bala$š^evi$ć&lt;/tt&gt;<br /> * morphologically analysing tokens, including those which only appear as part of a multiword<br /> ** contractions (1 token, &gt;1 word)<br /> ** compounds (1 token, &gt;1 word)<br /> * retokenisation for multiwords<br /> ** (&gt;1 token, 1 word)</div>

Francis Tyers