Difference between revisions of "Kazakh and Tatar/TODO"

From Apertium
Jump to navigation Jump to search
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== General TODO ==
== Goals ==


In both directions: apertium-kaz-tat has at least 15000 top stems, 95% coverage on all the corpora we have, and no more than 15% Word-Error-Rate on any randomly selected text. It has full coverage on '''Абай жолы. Бірінші кітап''' and disambiguates it fully with at least 90% of precision. There is a unified and documented testing framework to test morphotactics (specs for each [[Turkic_lexicon#Morphotactics|type II LEXICON]]), morphophonology, coverage, CG performance, regression and pending tests targeted mainly at transfer rules, a “gold standard” parallel corpus to measure the WER and to always have something to work on. Tests are fast (slow parts are decoupled). Testvoc is clean.
See [[Kazakh and Tatar/Work_plan]].


== Road map ==
# Declination of Tatar nouns ending with -и.
# <s>Set up <code>bidix-with-context.sh</code> script (see <code>apertium-kaz-tat/dev/bidix</code>; seems to be very useful, requires another script from spectie)</s>.
# <s>Add some of the short wikipedia-article-like texts I have for evaluation into <code>texts</code> (should be ~200 words).</s>
# Implement cont. class for compound/multiword nouns which already have possessive ending (<px3sp>), e.g. ''Қытай Халық Республикасы''.
## This continuation class should link only to CASE (but consider that some of them can have plural form: ''ишегаллары'').
# Add "ярты", "ярым" and "чирек" as numerals, but don't link them to common numerals cont. class.
# (Lexical selection rule): ''сондай-ақ'' > ''шулай-ук''
# <s>Fix roman numerals:
## add them to tat.lexc too;
## change <code>LEXICON NUM-ROMAN</code> to something like this: <code>%<num%>%<ord%>: # ; </code>.</s>
# Add transfer rule(s) to handle instrumental case of all parts-of-speech which are subject to substantivation, not only of nouns (this is one of the things which make testvoc results look bad)
# A separate cont.class for verbs which have causative forms ending with -дыр/-дер
# A "location-cases" cont. classes for some of the postpositions and location adverbs (e.g. "бире")
# '<code>Natinfl</code> cont. class in tat.lexc
# Fix "дыр<mod_ind>" thing (it doesn't pass bidix right now)
# '''Pronouns'''
## check cont. classes (note: if it looks like an overgeneration, and me is not sure about it, overgenerate in both lexc's)
## translate pronouns from kaz.lexc, add them to bidix and add equivalents into tat.lexc
## ^нигез/ни<prn><itg><px2pl><nom>
# '''Determiners'''
## "unify" cont. classes and tags
## add stems
# '''Adjectives'''
## personal clitics after adjectives are not implemented yet
# '''Translating between classes'''
##
# '''General stuff'''
## Copula suffixes: have for kazakh -ø = p3.sp, and for tatar -ø = p3.sp and -lAr = p3.pl. When translating kaz->tat there is no problem, when translating tat->kaz, pl->sp.


* coverage (more stems and better morphology)
== Phonology-related stuff ==
** <s>add "internationalisms" spectie has put in /dev</s>
** see also: [[Kazakh and Tatar/Remaining unanalysed forms]]
* constraint grammar
* transfer
* lexical selection


== General TODO ==
Might be twol, might not be, but JNW needs to go through this stuff and figure out the issues.

=== Kazakh ===

# <s>Current: <code>^миллион<num><subst><dat>$ --> миллионге</code> Should be: <code>^миллион<num><subst><dat>$ --> миллионға</code></s>
# <s>Current: <code>^сөйле<v><tv><coop><ger_past><loc>$ --> сөйлесгенде</code> Should be: <code>^сөйле<v><tv><coop><ger_past><loc>$ --> сөйлескенде</code></s>
# <s>Kazakh: <code>^ойна<v><tv><ifi><p1><pl>$ --> ойнадык</code> Should be: ''ойнадыҚ''</s>
# *жатқандығын
# безнекенеме (accusative case before clitics); безнекенгәме
# <s>*журналистерді - *журналистеріне - *журналистерді
#* something like <tt>т:0 <=> :с/:0 _ %{L%}:/:0</tt></s> (<tt>r40597</tt>)
# <s>(kaz) *Назарбаевтың</s> (<tt>r40594</tt>)
# АКШ-*тың НАТО-*ның
:: This is a problem with lexc, not twol —[[User:Firespeaker|Firespeaker]] 06:15, 20 August 2012 (UTC)
# <s>(kaz) *организмдер / организм<n><pl><nom> = организмдар</s> (<tt>r40597</tt>)
# (kaz) процесс, процесі/процессі, процесінің/процессінің
# <s>(kaz) автомобиль<n><attr>, *автомобильдер // автомобиль<n><pl><nom> = автомобильлер
:: As far as I can tell, автомобильдер is the most common form. The form автомобилдер also seems to be used, but doesn't look super formal, and автомобильлер seems to only be attested in "Kazakh" because Nissan seems to like to write in Noğay for its Kazakh-speaking audience. —[[User:Firespeaker|Firespeaker]] 06:10, 20 August 2012 (UTC)
::: The thing is that the form we are generating is автомобильлер. - [[User:Francis Tyers|Francis Tyers]] 07:08, 20 August 2012 (UTC)</s> (<tt>r40704</tt>)
# <s>^организмге/*организмге$ *организмнің *организмнен</s> (<tt>r40705</tt>)
# <s>*тарихынан *тарихы ...</s> (≤<tt>r40705</tt>)

=== Tatar ===

# (tat) generates ''укыу''
# (tat) generates ''айендә''
# Deletion of soft sign "ь" before vowels in Tatar (see comments at the end of the <code>apertium-tat/apertium-tat.tat.twol</code> file)
# (tat) ''*аенда'', generating ''музейе'' instead of correct ''музее''
# <code>apertium-tat$ echo "^йөр<v><iv><gpr_impf>$" | hfst-proc -g tat.autogen.hfst</code> >> <code>йөрә торган</code>

== Other ==
=== International vocabulary ===


* s/fut3/vol/
* <s>'''0 itself and numbers containing it aren't analyzed (in both directions)'''</s>
** <s>This is only true for the transducers in apertium-kaz-tat, apertium-kaz and apertium-tat ones work fine.</s>
* A number with a following . is analyzed incorrectly and therefore not generated:
** When apertium (not hfst-proc) is used, this is the case for any number at the end of the line, because deformatter puts a "." at the end of the sentence automatically.
<pre>
<pre>
/apertium-kaz$ echo "21." | hfst-proc kaz.automorf.hfst
*терроризмге *массивіндегі *террорлық *Факті
^21./21.<num>$
*кодекстің *терроризмге *Полицейлер *журналистерді «*АНТИТЕРРОРЛЫҚ »
*полицейлер *антитеррорлық *режим *полицейлер *журналистерді
*автоматты *автобустар *полицейлер *журналистеріне *сайттың

*технологиялар *компьютер *мобильді *техникаларға *интернет *объектілерін
*радиациялық *сантехник *проблемасы *веб-*сайттар *позитивті *алгебра

*коалициялық

*иммиграциялық *дипломатиялық *стратегиялық

*станциясында

</pre>
</pre>
* Make instrumental case to a clitical postposition, leaving only 6 cases which are the same both in Tatar and Kazakh (see [[http://wiki.apertium.org/wiki/Morphology_of_Kyrgyz_language#Cases_by_syntactic_function]] and the log from 12.03.2013 for reference)
** update the t1x files accordingly (i.e. get rid of the rules for handling instrumental case)
* Revise continuations of gerunds
* жігіт% %{М%}ен
* Declination of Tatar nouns ending with -и.
* A separate cont.class for verbs which have causative forms ending with -дыр/-дер
** Isn't this the default for {{tag|v}}{{tag|iv}} ?
* A "location-cases" cont. classes for some of the postpositions and location adverbs (e.g. "бире")
** What do you mean? —[[User:Firespeaker|Firespeaker]] 16:20, 6 February 2013 (UTC)
* [[Apertium-kaz-tat/Ideas_for_Disambiguation_Rules|Better disambiguation]]
* <code>көр%<v%>%<tv%>%<imp%>%<p2%>%<sg%>:гөр # ; ! "" Dir/LR</code> get's trimmed
* ма не - мыни thing
* handle gna_cond + DA<postadv> issue in lexc, not in CG
* Handle the sentences from the paper in transfer, not in CG
* Some nouns in Tatar (and Kazakh) lexc seem to be in NLEX and NLEX-RUS. This is fine for analysis, but which form is generated? There should be some <tt>! Dir/..</tt> filtering somewhere in there.
* Consider ''турындагы'' - should it still be tagged as postposition?
* How to handle verbs with inner inflection (sometimes a Kazakh verb is translated with a multiword and vice versa. E.g. ''әуреле > башын әйләндер'')
* a better default translation for Kazakh past.evid


=== Algorithm for checking dictionaries (as part of the testvocing) ===
===Proper nouns===


* Go through entries in bidix
<pre>
** Get rid of duplications, FIXME's, alternatively spelled variants (handling them in lexc instead)

* Look up bidix stems in lexc's and make sure every one of them is in there (and we don't loose any bit of coverage)
*Гонконгтан
* Try to get rid of FIXME's for stems in lexc's

* Have a quick look at continuation lexicons and make sure they follow standards (some of the lexicon names in tat.lexc in particular)
</pre>
* Run testvoc (either for each different class separately -- commenting out all other root lexicons -- or without modifying anything if it doesn't take forever)

* If a Tatar noun marked with 'Use/MT' is not used in kaz-tat.dix, get rid of it in tat.lexc
=== Discuss first ===

# There is only one formal form (<frm>) in Tatar, which can be both sg and plural. But in Kazakh there are two forms. Should I pretend as if in Tatar it *were* the same and duplicate the same form with a different tag or should I handle it in transfer?
# Consider ''турындагы'' - should it still be tagged as postposition?
# How to handle verbs with inner inflection (sometimes a Kazakh verb is translated with a multiword and vice versa. E.g. ''әуреле > башын әйләндер'')
----

Part-of-speech related TODO's and DONE's can be found here:

* [[Kazakh and Tatar/Postadvebs|/Postadverbs]]
* [[Kazakh and Tatar/Postpositions|/Postpositions]]

To run tests, use <code>aq-regtest</code> utility from [[Apertium-quality]] tools. E.g. <pre>aq-regtest -d . kaz-tat http://wiki.apertium.org/wiki/Special:Export/Kazakh_and_Tatar/Postadvebs</pre>

== Done ==

; But keep an eye on this

* Numerals
** kaz <num><subst>(<px3>) in fractions<ref>Currently whether it is in fractions or not is not taken into account</ref> = tat <num><subst>(<px3>)
** kaz <num><coll><advl> = tat <num><coll>
** kaz <num><coll><subst> = tat <num><subst>


== Notes ==
== Notes ==
Line 123: Line 60:
* [[Kazakh and Tatar/Pending tests]]
* [[Kazakh and Tatar/Pending tests]]
* [[Kazakh and Tatar/Regression tests]]
* [[Kazakh and Tatar/Regression tests]]

[[Category:Kazakh and Tatar|*]]
[[Category:TODO lists]]

Latest revision as of 21:20, 31 August 2015

Goals[edit]

In both directions: apertium-kaz-tat has at least 15000 top stems, 95% coverage on all the corpora we have, and no more than 15% Word-Error-Rate on any randomly selected text. It has full coverage on Абай жолы. Бірінші кітап and disambiguates it fully with at least 90% of precision. There is a unified and documented testing framework to test morphotactics (specs for each type II LEXICON), morphophonology, coverage, CG performance, regression and pending tests targeted mainly at transfer rules, a “gold standard” parallel corpus to measure the WER and to always have something to work on. Tests are fast (slow parts are decoupled). Testvoc is clean.

Road map[edit]

General TODO[edit]

  • s/fut3/vol/
  • 0 itself and numbers containing it aren't analyzed (in both directions)
    • This is only true for the transducers in apertium-kaz-tat, apertium-kaz and apertium-tat ones work fine.
  • A number with a following . is analyzed incorrectly and therefore not generated:
    • When apertium (not hfst-proc) is used, this is the case for any number at the end of the line, because deformatter puts a "." at the end of the sentence automatically.
/apertium-kaz$ echo "21." | hfst-proc kaz.automorf.hfst 
^21./21.<num>$
  • Make instrumental case to a clitical postposition, leaving only 6 cases which are the same both in Tatar and Kazakh (see [[1]] and the log from 12.03.2013 for reference)
    • update the t1x files accordingly (i.e. get rid of the rules for handling instrumental case)
  • Revise continuations of gerunds
  • жігіт% %{М%}ен
  • Declination of Tatar nouns ending with -и.
  • A separate cont.class for verbs which have causative forms ending with -дыр/-дер
    • Isn't this the default for <v><iv> ?
  • A "location-cases" cont. classes for some of the postpositions and location adverbs (e.g. "бире")
    • What do you mean? —Firespeaker 16:20, 6 February 2013 (UTC)
  • Better disambiguation
  • көр%<v%>%<tv%>%<imp%>%<p2%>%<sg%>:гөр # ; ! "" Dir/LR get's trimmed
  • ма не - мыни thing
  • handle gna_cond + DA<postadv> issue in lexc, not in CG
  • Handle the sentences from the paper in transfer, not in CG
  • Some nouns in Tatar (and Kazakh) lexc seem to be in NLEX and NLEX-RUS. This is fine for analysis, but which form is generated? There should be some ! Dir/.. filtering somewhere in there.
  • Consider турындагы - should it still be tagged as postposition?
  • How to handle verbs with inner inflection (sometimes a Kazakh verb is translated with a multiword and vice versa. E.g. әуреле > башын әйләндер)
  • a better default translation for Kazakh past.evid

Algorithm for checking dictionaries (as part of the testvocing)[edit]

  • Go through entries in bidix
    • Get rid of duplications, FIXME's, alternatively spelled variants (handling them in lexc instead)
  • Look up bidix stems in lexc's and make sure every one of them is in there (and we don't loose any bit of coverage)
  • Try to get rid of FIXME's for stems in lexc's
  • Have a quick look at continuation lexicons and make sure they follow standards (some of the lexicon names in tat.lexc in particular)
  • Run testvoc (either for each different class separately -- commenting out all other root lexicons -- or without modifying anything if it doesn't take forever)
  • If a Tatar noun marked with 'Use/MT' is not used in kaz-tat.dix, get rid of it in tat.lexc

Notes[edit]


See also[edit]

Retrieved from "https://wiki.apertium.org/w/index.php?title=Kazakh_and_Tatar/TODO&oldid=54058"