Alphabet

The <alphabet> part of lttoolbox dictionaries is used to specify which characters are considered possible parts of words as opposed to "blank" chars. Its main effect is on tokenisation of unknown words, since non-alphabet characters may still be part of a known word.

Example

Say your deu.dix looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<dictionary>
	<alphabet>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz</alphabet>
	<sdefs>
		<sdef n="prn" 	c="Pronoun"/>
		<sdef n="pr" 	c="Preposition"/>
	</sdefs>
	<section id="main" type="standard">
		<e> <p> <l>es</l> <r>es<s n="prn"/></r> </p> </e>
		<e> <p> <l>über</l> <r>über<s n="pr"/></r> </p> </e>
	</section>
</dictionary>

We compile it:

$ lt-comp lr deu.dix deu.bin
main@standard 4 3

And it works as expected for "es":

$ echo es | lt-proc deu.automorf.bin
^es/es<prn>$

Given a word not in the dictionary, but composed of alphabetic chars, we get an unknown-word analysis:

$ echo wei | lt-proc deu.automorf.bin
^wei/*wei$

But if that unknown word contains a non-alphabetic char, the unknown word analysis will be split on that char (which will be considered a blank):

$ echo große | lt-proc deu.automorf.bin
^gro/*gro$ß^e/*e$

This is as if you'd input "gro e".

If a non-alphabetic is part of a known analysis, it'll work just fine:

$ echo über grün | lt-proc deu.automorf.bin
^über/über<pr>$ ^gr/*gr$ü^n/*n$

(über is in the dictionary, grün is not)

Most of the time, this isn't very problematic, but it may become a problem if the splitting leads to wrong analyses. Here the splitting leads to an analysis of the pronoun "es", which is wrong and possibly very confusing in the translated output:

$ echo weißes | lt-proc deu.automorf.bin
^wei/*wei$ß^es/es<prn>$

Wishlist

• Ability to specify Unicode ranges