Alphabet - Revision history

Unhammer: /* Example */

2014-04-28T08:52:41Z

Example

Unhammer at 08:45, 28 April 2014

2014-04-28T08:45:04Z

Unhammer: /* Example */

2014-04-28T08:32:18Z

Example

Unhammer: /* Example */

2014-04-28T08:32:00Z

Example

Unhammer at 08:31, 28 April 2014

2014-04-28T08:31:09Z

Unhammer: Created page with "The '''<alphabet>''' part of lttoolbox dictionaries is used to specify which characters are considered possible parts of words as opposed to "blank" chars. Its main ..."

2014-04-28T08:28:43Z

Created page with "The '''<alphabet>''' part of lttoolbox dictionaries is used to specify which characters are considered possible parts of words as opposed to "blank" chars. Its main ..."

New page

The '''<alphabet>''' part of [[lttoolbox]] dictionaries is used to specify which characters are considered possible parts of words as opposed to "blank" chars. Its main effect is on tokenisation of ''unknown'' words, since non-alphabet characters may still be part of a ''known'' word.

Say your deu.dix looks like this:
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<dictionary>
<alphabet>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz</alphabet>
<sdefs>
<sdef n="prn" c="Pronoun"/>
<sdef n="pr" c="Preposition"/>
</sdefs>
<section id="main" type="standard">
<e> <p> <l>es</l> <r>es<s n="prn"/></r> </p> </e>
<e> <p> <l>über</l> <r>über<s n="pr"/></r> </p> </e>
</section>
</dictionary>
</pre>
We compile it:
<pre>
$ lt-comp lr deu.dix deu.bin
main@standard 4 3
</pre>

And it works as expected for "es":
<pre>
$ echo es | lt-proc deu.automorf.bin
^es/es<prn>$
</pre>

Given a word not in the dictionary, but composed of alphabetic chars, we get an unknown-word analysis:
<pre>
$ echo wei | lt-proc deu.automorf.bin
^wei/*wei$
</pre>

But if that unknown word contains a non-alphabetic char, the unknown word analysis will be split on that char (which will be considered a blank):
<pre>
$ echo große | lt-proc deu.automorf.bin
^gro/*gro$ß^e/*e$
</pre>
This is as if you'd input "gro e".

But if a non-alphabetic is specified as part of a known analysis, it'll work just fine:
<pre>
$ echo über grün | lt-proc deu.automorf.bin
^über/über<pr>$ ^gr/*gr$ü^n/*n$
</pre>
(über is in the dictionary, grün is not)

==Wishlist==

* Ability to specify Unicode ranges

@@ Line 55: / Line 55: @@
 ^wei/*wei$ß^es/es<prn>$
 </pre>
+==Can't we just put everything into the alphabet then?==
+Too many alphabetics can lead to problems too, e.g. if you put space characters in your alphabet, you won't get any tokenisation on spaces! If we put a space into the alphabet of our example, even known words will not get analyses if they have spaces around them:
+<pre>
+$ echo es es | lt-proc deu.automorf.bin
+^es es/*es es$
+</pre>
+(But see also [[Inconditional section]].)
 ==Wishlist==

@@ Line 59: / Line 59: @@
 * Ability to specify Unicode ranges
+[[Category:Documentation in English]]
+[[Category:Writing dictionaries]]

@@ Line 50: / Line 50: @@
-Most of the time, this isn't very problematic, but it may become a problem if the splitting leads to wrong analyses. Here the splitting leads to an analysis of the pronoun "es", which is wrong and possibly very confusing in the translated output:
+Most of the time, missing alphabetics isn't very problematic, but it may become a problem if the splitting leads to wrong analyses. Here the splitting leads to an analysis of the pronoun "es", which is wrong and possibly very confusing in the translated output:
 <pre>
 $ echo weißes | lt-proc deu.automorf.bin

@@ Line 42: / Line 42: @@
 This is as if you'd input "gro e".
-But if a non-alphabetic is specified as part of a known analysis, it'll work just fine:
+If a non-alphabetic is part of a ''known'' analysis, it'll work just fine:
 <pre>
 $ echo über grün | lt-proc deu.automorf.bin
@@ Line 55: / Line 55: @@
 ^wei/*wei$ß^es/es<prn>$
 </pre>
 ==Wishlist==

@@ Line 1: / Line 1: @@
 The '''&lt;alphabet&gt;''' part of [[lttoolbox]] dictionaries is used to specify which characters are considered possible parts of words as opposed to "blank" chars. Its main effect is on tokenisation of ''unknown'' words, since non-alphabet characters may still be part of a ''known'' word.
+==Example==
 Say your deu.dix looks like this:
 <pre>
@@ Line 47: / Line 48: @@
 </pre>
 (über is in the dictionary, grün is not)
+Most of the time, this isn't very problematic, but it may become a problem if the splitting leads to wrong analyses. Here the splitting leads to an analysis of the pronoun "es", which is wrong and possibly very confusing in the translated output:
+<pre>
+$ echo weißes | lt-proc deu.automorf.bin
+^wei/*wei$ß^es/es<prn>$
+</pre>