Apertium-kaz - Revision history

Ilnar.salimzyan: s/sourceforge link/github link/

2018-03-08T17:45:48Z

s/sourceforge link/github link/

Firespeaker: /* To use */

2017-12-10T20:00:07Z

To use

Firespeaker: /* To use */

2017-12-10T19:58:18Z

To use

Ilnar.salimzyan: /* Additional tags */

2017-10-30T20:31:21Z

Additional tags

Ilnar.salimzyan: /* Developers */

2017-10-30T20:25:50Z

Developers

Ilnar.salimzyan at 20:19, 30 October 2017

2017-10-30T20:19:30Z

Ilnar.salimzyan: /* Additional tags */

2017-10-30T19:59:16Z

Additional tags

Ilnar.salimzyan: /* Additional tags */

2017-10-30T19:58:06Z

Additional tags

Ilnar.salimzyan: some more docs

2017-10-30T19:54:42Z

some more docs

Show changes

Ilnar.salimzyan: /* Developers */

2017-10-30T10:28:51Z

Developers

@@ Line 18: / Line 18: @@
 == Installation ==
-'''Apertium-kaz''' is currently located in [https://svn.code.sf.net/p/apertium/svn/languages/apertium-kaz/ languages/apertium-kaz].
+'''Apertium-kaz''' is currently located in [https://github.com/apertium/apertium-kaz languages/apertium-kaz].
 === To use ===

@@ Line 22: / Line 22: @@
 === To use ===
 * [[Installation#Installing:_a_summary|Install Apertium core tools]].
-* You can install apertium-kaz from the same repo as the core tools, but what's recommended is to get it from the repo above and compile by running
+* You can install apertium-kaz from the same repo as the core tools, but it doesn't give you access to the source code.  For developer access, it is recommended to get the code from the repo above and compile it by running the following commands ''within the apertium-kaz directory'' that you checked out.
 *: <code>./autogen.sh; make</code>
 * Now test it by running some Kazakh words through the analyser:

@@ Line 24: / Line 24: @@
 * You can install apertium-kaz from the same repo as the core tools, but what's recommended is to get it from the repo above and compile by running
 *: <code>./autogen.sh; make</code>
-* Now test it by running some Kazakh words through the analyser it:
+* Now test it by running some Kazakh words through the analyser:
 *: <code>echo "бірнеше қазақша сөздер" | apertium -d . kaz-morph</code>
 ** You should get some analyses, like this:
 *: <code>^бірнеше/бірнеше{{tag|det}}{{tag|qnt}}$ ^қазақша/қазақша{{tag|adv}}/қазақша{{tag|n}}{{tag|nom}}/қазақша{{tag|n}}{{tag|attr}}/қазақша{{tag|n}}{{tag|nom}}+е{{tag|cop}}{{tag|aor}}{{tag|p3}}{{tag|pl}}/қазақша{{tag|n}}{{tag|nom}}+е{{tag|cop}}{{tag|aor}}{{tag|p3}}{{tag|sg}}$ ^сөздер/сөз{{tag|n}}{{tag|pl}}{{tag|nom}}/сөз{{tag|n}}{{tag|pl}}{{tag|nom}}+е{{tag|cop}}{{tag|aor}}{{tag|p3}}{{tag|pl}}/сөз{{tag|n}}{{tag|pl}}{{tag|nom}}+е{{tag|cop}}{{tag|aor}}{{tag|p3}}{{tag|sg}}$^./.<sent>$</code>
+** ''Note that <code>.</code> specifies the directory where the compiled transducer is.  The line above assume you're running the command from within the <code>apertium-kaz/</code> directory.''
 ==== Dependency tree ====

@@ Line 178: / Line 178: @@
 In the examples above, if you don't mark either of the stems with <code>Dir/LR</code>, then the Kazakh generator, (if we personify it a bit) given a string like "^сұхбат<n><nom>$ for input, won't know which surface form to choose and will output both, separated with a slash: сұхбат/сұқбат.
-As the name suggests, '''<code>Dir/RL</code>''' has the meaning opposite to <code>Dir/LR</code>: 'generate this surface form, but do not analyse it'. You won't see it much in a lexc file and almost certainly won't need to mark as a stem you add as Dir/RL. Here is an example though:
+As the name suggests, '''<code>Dir/RL</code>''' has the meaning opposite to <code>Dir/LR</code>: 'generate this surface form, but do not analyse it'. You won't see it much in a lexc file and almost certainly won't need to mark a stem you add as Dir/RL. Here is an example though:
 <pre>

@@ Line 72: / Line 72: @@
 # what the stem of the word is (to be exact, what the left-hand side and the right-hand side of the the entry should be),
 # whether or not that stem is already in apertium-kaz, and
-# (if it isn't or it isn't something that is needed) which continuation lexicon (read: paradigm) you should assign the stem to.
+# (if it isn't or it isn't analysed as something that you expect) which continuation lexicon (read: paradigm) you should assign the stem to.
 Here is an example of a word already in the apertium-kaz.kaz.lexc file:

@@ Line 155: / Line 155: @@
 ==== Adverbs ====
 * If you want to add an adverb, first think whether the word is really an adjective that can be used like an adverb.  If this is the case, then add it as an adjective in the appropriate adjective class that can take the {{tag|advl}} tag.  In the bidix, you'll want to translate the {{tag|adj}} and the {{tag|adj}}{{tag|advl}} forms differently.
+=== Additional tags ===
+In a .lexc file, after the '!' you will also see <code>Dir/LR</code>, <code>Dir/RL</code>, <code>Err/Orth</code> and <code>Use/MT</code> comments. The meaning of them is as follows:
+'''<code>Dir/LR</code>''' means: analyse this surface form, but don't generate it. Here is a good example:
+<pre>
+сұхбат:сұқбат N1 ; ! "conversation/interview" Dir/LR
+сұхбат:сұхбат N1 ; ! "conversation/interview"
+</pre>
+In other words, <code>Dir/LR</code> marks alternative spellings of a word. If the alternative spelling isn't just alternative, but actually erroneous (but occurs quite commonly so that you want to support it), it is marked with the '''<code>Err/Orth</code>''' tag:
+<pre>
+орын:ор%{y%}н N1 ; ! "place,seat"
+орын:орын N1 ; ! "place,seat"  ! Dir/LR ! Err/Orth
+</pre>
+"Орыны" for example, is considered erroneous spelling of "орын<n><px3sp><nom>". Such markings will allow us to produce better spell checkers.
+In the examples above, if you don't mark either of the stems with <code>Dir/LR</code>, then the Kazakh generator, (if we personify it a bit) given a string like "^сұхбат<n><nom>$ for input, won't know which surface form to choose and will output both, separated with a slash: сұхбат/сұқбат.
+As the name suggests, '''<code>Dir/RL</code>''' has the meaning opposite to <code>Dir/LR</code>: 'generate this surface form, but do not analyse it'. You won't see it much in a lexc file and almost certainly won't need to mark as a stem you add as Dir/RL. Here is an example though:
+<pre>
+да:%~да CC ; ! "also" Dir/RL
+</pre>
+The conjunction ^да<cnj$ gets generated as "~да". This is necessary for a somewhat hacky way of handling the vowel harmony (read: making sure that the "да" gets rendered as "де" when the preceding word has front vowels) in cases where the standard way of handling the vowel harmony (read: [[twol]]) fails because the preceding word is unknown.
+'''Use/MT''' (at least, in its original usage) marks (compound) words which are needed for translation, but probably shouldn't be in a "vanilla" Kazakh transducer:
+<pre>
+қайда% болса% сонда:қайда% болса% сонда PRON-IND ; ! "anywhere" Use/MT
+</pre>
+It has been also used to mark words which the person who added them wasn't sure how to classify. Such words will be reviewed later.
 === Full inventory of lexicons the stems can be linked to ===
@@ Line 218: / Line 256: @@
 * ADV-WITH-KI-I
 * ADV-LANG
-=== Additional tags ===
-In a .lexc file, after the '!' you will also see <code>Dir/LR</code>, <code>Dir/RL</code>, <code>Err/Orth</code> and <code>Use/MT</code> comments. The meaning of them is as follows:
-'''<code>Dir/LR</code>''' means: analyse this surface form, but don't generate it. Here is a good example:
-<pre>
-сұхбат:сұқбат N1 ; ! "conversation/interview" Dir/LR
-сұхбат:сұхбат N1 ; ! "conversation/interview"
-</pre>
-In other words, <code>Dir/LR</code> marks alternative spellings of a word. If the alternative spelling isn't just alternative, but actually erroneous (but occurs quite commonly so that you want to support it), it is marked with the '''<code>Err/Orth</code>''' tag:
-<pre>
-орын:ор%{y%}н N1 ; ! "place,seat"
-орын:орын N1 ; ! "place,seat"  ! Dir/LR ! Err/Orth
-</pre>
-"Орыны" for example, is considered erroneous spelling of "орын<n><px3sp><nom>". Such markings will allow us to produce better spell checkers.
-In the examples above, if you don't mark either of the stems with <code>Dir/LR</code>, then the Kazakh generator, (if we personify it a bit) given a string like "^сұхбат<n><nom>$ for input, won't know which surface form to choose and will output both, separated with a slash: сұхбат/сұқбат.
-As the name suggests, '''<code>Dir/RL</code>''' has the meaning opposite to <code>Dir/LR</code>: 'generate this surface form, but do not analyse it'. You won't see it much in a lexc file and almost certainly won't need to mark as a stem you add as Dir/RL. Here is an example though:
-<pre>
-да:%~да CC ; ! "also" Dir/RL
-</pre>
-The conjunction ^да<cnj$ gets generated as "~да". This is necessary for a somewhat hacky way of handling the vowel harmony (read: making sure that the "да" gets rendered as "де" when the preceding word has front vowels) in cases where the standard way of handling the vowel harmony (read: [[twol]]) fails because the preceding word is unknown.
-'''Use/MT''' (at least, in its original usage) marks (compound) words which are needed for translation, but probably shouldn't be in a "vanilla" Kazakh transducer:
-<pre>
-қайда% болса% сонда:қайда% болса% сонда PRON-IND ; ! "anywhere" Use/MT
-</pre>
-It has been also used to mark words which the person who added them wasn't sure how to classify. Such words will be reviewed later.
 [[Category:Tools]]

@@ Line 64: / Line 64: @@
 == Developers ==
+=== Guidelines for adding stems ===
-We have several language pairs involving Kazakh, and in every pair there is a lexc, twol and rlx file for this language. But we don't work on these files directly. Instead, we edit <code>kaz.lexc</code>, <code>kaz.twol</code> and <code>kaz.rlx</code> files located in '''apertium-kaz''', and then import these files to the language pair directories using a script (<code>update-morphs.bash</code> in language-pair directories). This script merely copies twol and rlx files, since they don't have to be tweaked to a particular language pair, but "trimms" the lexc file leaving only that stems in it, which are also found in the bilingual dictionary of the pair importing is made to.
+==== An overview of the process ====
-For further details on how these works and a step-by-step guide (taking the Kazakh-Tatar pair as an example), see [[Kazakh_and_Tatar#Development_workflow]].
+If you see that a wordform is not supported by apertium-kaz and you want to add it, you have to figure out three things:
-A natural consequence of this approach is that any change made to '''apertium-kaz''' will affect all other Kazakh-to-X or X-to-Kazakh language pairs. This requires some rules to be followed. Here are some of them.
-# Do not change tags or order of tags and don't do any other change wich certainly will break things in other pairs without discussing it first on the apertium-turkic mailing list (or notifying others via the list if you are absolutely sure that this change was necessary (e.g. it was discussed earlier and hence was due));
+) what the stem of the word is (and also the right-hand side in the entry)
-# If you encounter a word in the lexc which seems to be miscategorized, or some multiword which in reality is a combination of two lexemes (like ''барлық жерде'' from the example above), do not delete them! Mark them with <code>Use/MT</code> instead.
+) which continuation lexicon (read: paradigm) you should assign to it
-# Write descriptive commit messages.
+) whether or not that stem is already in apertium-kaz
+Here is an example of a word already in apertium-kaz.kaz.lexc file:
+кітап:кітап N1 ; ! "book"
+As in this example, in most cases, the left hand-side and the right-hand side of the entry will be the same. The left-hand side might be referred to as 'stem'. Continuation lexicon is N1. What comes after '!' are comments. Glosses are a good thing to have, but technically it is only comment, and thus optional.
+An example where the left and right hand sides are not the same:
+күн% тәртібі:күн% тәртіп N-COMPOUND-PX ; ! ""
+This has been done so so that forms like "күн тәртіптері" can also be analyzed.
+The example above also shows that spaces in a word have to be escaped with %. So is the hyphen sign:
+мән%-жай:мән%-жай N1 ; ! ""
-=== Guidelines for adding stems ===
 ==== General ====
-* Before adding a stem, be sure it does not already exist in lexc.
+* Before adding a stem, be sure it does not already exist in lexc. A good way to do that is to look up stem(s) you want to add with <code>lt-proc kaz.automorf.bin</code>. In some cases, you'll see that the stem isn't analysed at all:
+^foo/*foo$
+In some cases, it will be analysed, but as something else than what you want to add it as:
+^Жол/жол<adj>$ ^жөндеуші/жөнде<v><tv><gpr_pot>$^./.<sent>$
+(Assuming that you want to add "Жол жөндеуші as a company name, which it happens to be).
+Another, probably more relevant example:
+<pre>
+apertium-kaz$ echo "қабылдау" | apertium -d . kaz-tagger
+^қабылдау/қабылда<v><tv><ger><nom>$^./.<sent>$
+</pre>
+I'm pretty sure that you shouldn't add қабылдау as a noun. Or, rather, that you wouldn't after you see the above analysis.
+In the third case, you will see that the stem gets the analysis you expect, but you'd seen that some of it wordforms were not analysed. This means that either there is a problem with the phonology part, or you've discovered some affix currently not supported by apertium-kaz. Both issues have to be documented/reported (the simplest way would be just to add an 'ISSUES' file to apertium-kaz and commit it).
 * Provide a commit message saying what you did.  At a bare minimum, "adding more stems" is okay, but "a" or "ф" is not.  Try to be more informative though; e.g. "added stems from story, mostly NP-TOP and NP-ANT" or similar.
 * Many stems exhibit a voicing alternation like п/б, к/г, қ/ғ.  This is processed automatically by twol, but these stems ''must'' be added with the ''voiceless'' consonant (п, к, қ), e.g <code>тақ:тақ V-TV ;</code>
@@ Line 82: / Line 119: @@
 ** Words that are commonly written in both forms (e.g., орнында and орынында) need special treatment: add <code>! Dir/LR</code> after the form that should not be generated (i.e., the form that is the non-normative version), and add <code>! Err/Orth</code> after it too if it should be considered a spelling mistake.
 * Any changes to continuation classes should be discussed on the apertium-turkic mailing list.
+Most likely, a word not covered by apertium-kaz already will be an open class word. Below are some comments on the open-class word lexicons.
 ==== Verbs ====
 * Categorise correctly according to IV or TV status:
 ** IV = intransitive verbs; TV = transitive verbs
-** If the verb can take a direct object with -НЫ, then it's not IV; otherwise it is
+** If the verb can take a direct object with -НЫ, then it's not IV; otherwise it is TV
 ** For phrasal verbs (e.g,. "қабыл ал", "пайда бол", "мойынға ал"), do not categorise it according to its elements; instead treat it as a single verb (TV, IV, TV).
 * There should be no infinitival final -у or -ю.  It is best to take the part of the verb before -GAн or -DI in those forms.
@@ Line 105: / Line 144: @@
 ==== Adjectives ====
 * The basic categorisation of adjectives depends on whether it takes comparative morphology (-ЫрАҚ), can be substantivised (acts like a noun), and/or can be adverbialised (acts like an adverb).  Be sure to put the adjective in the right category according what those categories allow.
 * If you're adding an adjective that can also be used as a noun, think whether it's actually an adjective or actually a noun and add it to the right category.  You'll want to subcategorise it correctly so that e.g. if it's an adjective it can also take the {{tag|subst}} tag.