Turkic languages

From Apertium
Jump to navigation Jump to search

The Turkic languages include Turkish, Azeri, Uzbek, Uyghur, Kazakh, Kyrgyz, Tatar, Chuvash, Turkmen, Bashkir, Sakha, Crimean Tatar and several dozen other languages. The languages are related with varying levels of mutual intelligibility. A team called Apertium Turkic aims to enhance MT coverage of these languages within Apertium.

The ultimate master plan involves generating independent finite-state transducers for each language, and then making individual dictionaries and transfer rules for every pair. The current status of these goals is listed below.

Status

The ultimate goal is to have multi-purpose transducers and annotated corpora (i.e. treebanks) for a variety of Turkic languages. These can then be paired for X→Y translation with the addition of a CG for language X and transfer rules / dictionary for the pair X→Y. Below is listed development progress for each language's transducers and dictionary pairs.

Transducers

See also: Turkic lexicon and Languages

Once a transducer has ~80% coverage on a range of medium-large corpora we can say it is "working". Over 90% and it can be considered to be "production".

name Language native name ISO 639 formalism state stems coverage location primary authors
-1 -3
apertium-kaz Kazakh қазақ тілі kk kaz HFST (lexc+twol) production 36,595 ~94.5% apertium-kaz (languages) Ilnar, Jonathan, Fran, Aida, Beknazar, Nathan
apertium-tat Tatar татар теле tt tat HFST (lexc+twol) production 55,702 ~91% apertium-tat (languages) Ilnar, Fran, Jonathan, Röstäm
apertium-kir Kyrgyz кыргыз тили ky kir HFST (lexc+twol) working 14,424 ~90.4% apertium-kir (languages) Jonathan, Mirlan, Fran, Qantörö
apertium-tyv Tuvan Тыва дыл - tyv HFST (lexc+twol) working 11,695 ~92.7% apertium-tyv (languages) Fran, Алдынай, Aziyana, Jonathan
apertium-tur Turkish Türk dili tr tur HFST (lexc+twol) development 17,221 ~87.3% apertium-tur (incubator) Fran, Gianluca, Jonathan
apertium-crh Crimean Tatar qırımtatar tili - crh HFST (lexc+twol) development 11,757 ~85.4% apertium-crh (languages) Fran, Jonathan
apertium-chv Chuvash чӑваш чӗлхи cv chv HFST (lexc+twol) development 10,267 ~85% apertium-chv (languages) Hèctor
apertium-sah Sakha Саха тыла - sah HFST (lexc+twol) development 11,531 ~89.6% apertium-sah (languages) Sardana Ivanova, Fran, Jonathan, Christopher Straughn
apertium-kum Kumyk къумукъ тил - kum HFST (lexc+twol) development 4,918 ~90.2% apertium-kum (languages) Fran, Jonathan
apertium-kaa Karakalpak Qaraqalpaq tili - kaa HFST (lexc+twol) development 25,545 ~86.1% apertium-kaa (languages) Beknazar, Fran, Jonathan
apertium-uzb Uzbek o'zbek tili uz uzb HFST (lexc+twol) development 34,470 ~82.9% apertium-uzb (languages)
apertium-krc Karachay-Balkar Къарачай-Малкъар тил - krc HFST (lexc+twol) prototype 7,895 ~89.7% apertium-krc (incubator) Fran
apertium-bak Bashkir башҡорт теле ba bak HFST (lexc+twol) development 2,827 ~66% apertium-bak (languages) Fran, Jonathan, Ilnar, Milli
apertium-nog Nogay Ногай тили - nog HFST (lexc+twol) development 1,385 ~81.4% apertium-nog (languages) Fran, Jonathan
apertium-gag Gagauz Gagauz dili - gag HFST (lexc+twol) prototype 3,143 ~78% apertium-gag (incubator) Fran Sevilay
apertium-tuk Turkmen Türkmençe tk tuk HFST (lexc+twol) development 2,988 ~70.7% apertium-tuk (languages) Fran
apertium-uig Uyghur ئۇيغۇر تىلى ug uig HFST (lexc+twol) prototype 17,585 ~54.2% apertium-uig (incubator) Jonathan, Märdan, Fran
apertium-alt Altay Алтай тили - alt HFST (lexc+twol) prototype 284 ~52.27% apertium-alt (incubator) Jonathan, D Tokmashev
apertium-kjh Khakas Хакас тілі - kjh HFST (lexc+twol) prototype 293 ~39.3% apertium-kjh (incubator) Fran
apertium-ota Ottoman Turkish لسان عثمانى‎ - ota HFST (lexc+twol) prototype 75 ~10.8% apertium-ota (incubator) Fran
apertium-aze Azerbaijani Azərbaycan dili az aze HFST (lexc+twol) prototype 11,120 apertium-aze (incubator) Gianluca

Annotated corpora

name Language native name ISO 639 disambiguated dep-annotated avg. length location primary authors
-2 -3 sent tokens sent tokens
apertium-kaz Kazakh қазақ тілі kk kaz - - 403 5086 apertium-kaz (languages) Jonathan, Fran
apertium-tyv Tuvan Тыва дыл - tyv - - 115 639 apertium-tyv (languages) Jonathan, Fran
apertium-crh Crimean Tatar qırımtatar tili - crh - - 150 855 5.7 apertium-crh (languages) Fran
apertium-krc Karachay-Balkar - krc - - 17 116 6.8 Fran
apertium-uig Uyghur ئۇيغۇر تىلى - uig 123 951 27 191 7.1 apertium-uig (incubator) Fran

Turkic languages by subgroup

Pairs

Some Turkic languages that are particularly similar to one another (and hence have varying degrees of mutual intelligibility) include those in the following list:

Chuvash is very distant from other Turkic languages and is not even partially mutually intelligible with any of them.

Table of existing pairs

See also: Turkic-Turkic translator

Text in italics denotes prototype language pairs in the incubator. Regular text denotes a developing language pair in nursery, while text in bold denotes a stable well-working language pair in trunk, and text in bold and italics denotes a pair in staging. Bidix stems as counted with dixcounter are displayed below.

tur aze tuk uzb kir kaz tat bak chv nog kaa kum uig krc crh gag sah tyv kjh
tur - tur-aze
8,194		 tuk-tur
3,387		 tur-uzb
13,086		 tur-kir
7,123		 kaz-tur
7,526		 tur-tat
3,317		 chv-tur
31,930		 uig-tur
10,159		 crh-tur
7,106
aze tur-aze
8,194		 -
tuk tuk-tur
3,387		 -
uzb tur-uzb
13,086		 - kir-uzb
268		 uzb-kaa
24,114
kir tur-kir
7,123		 kir-uzb
268		 - kaz-kir
8,174		 tat-kir
kaz kaz-tur
7,526		 kaz-kir
8,174		 - kaz-tat
9,956		 nog-kaz
9		 kaz-kaa
5,408		 kaz-kum
561		 kaz-uig
2,728		 kaz-sah
65		 kaz-tyv
124
tat tur-tat
3,317		 tat-kir
 kaz-tat
9,956		 - tat-bak
57,710		 chv-tat
29,456
bak tat-bak
57,710		 -
chv chv-tur
31,930		 chv-tat
29,456		 -
nog nog-kaz
9		 -
kaa uzb-kaa
24,114		 kaz-kaa
5,408		 -
kum kaz-kum
561		 -
uig uig-tur
10,159		 kaz-uig
2,728		 -
krc -
crh crh-tur
7,106		 -
gag -
sah kaz-sah
65		 -
tyv kaz-tyv
124		 -
kjh -
eng tr-en
171		 eng-kir
465		 eng-kaz
16,931		 tat-eng
13,653		 sah-eng
?
epo eo-tr
1,500
fin tur-fin
21
khk khk-kaz
134
rus kaz-rus
9,925		 tat-rus
5,999		 cv-ru
75

Roadmap

  • Stable release of apertium-kaz
  • Stable release of apertium-tat
  • Stable release of apertium-kaz-tat
  • Rework apertium-kir to match new standards
  • Bring apertium-bak up to date (based on apertium-tat)
  • Expand apertium-tat-bak
  • Beta release of apertium-kaz-kir
  • Beta release of apertium-tur-kir
  • Expand apertium-tuk
  • Expand apertium-chv
  • Basic transducers for:

Getting involved

We have a work plan for developing Turkic-Turkic translators and are working on a how-to for building a Turkic lexicon. Please come talk to us on IRC or contact us on the apertium-turkic mailing list.

Samples

Article 1 of the Universal Declaration of Human Rights:

All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.

Language Text
Kazakh Барлық адамдар тумысынан азат және қадір‐қасиеті мен кұқықтары тең болып дүниеге келеді. Адамдарға ақыл‐парасат, ар‐ождан берілген, сондықтан олар бір‐бірімен туыстық, бауырмалдық қарым‐қатынас жасаулары тиіс.
Tatar Барлык кешеләр дә азат һәм үз абруйлары һәм хокуклары ягыннан тиң булып туалар. Аларга акыл һәм вөҗдан бирелгән һәм бер-берсенә карата туганнарча мөнәсәбәттә булырга тиешләр.
Kyrgyz Бардык адамдар өз беделинде жана укуктарында эркин жана тең укуктуу болуп жаралат. Алардын аң‐сезими менен абийири бар жана бири‐бирине бир туугандык мамилекылууга тийиш.
Turkish Bütün insanlar hür, haysiyet ve haklar bakımından eşit doğarlar. Akıl ve vicdana sahiptirler ve birbirlerine karşı kardeşlik zihniyeti ile hareket etmelidirler.
Azerbaijani Bütün insanlar ləyaqət və hüquqlarına görə azad və bərabər doğulurlar. Onların şüurları və vicdanları var və bir-birlərinə münasibətdə qardaşlıq ruhunda davranmalıdırlar.
Gagauz İnsannar hepsi duuêrlar serbest hem birtakım kendi kıymetindä hem haklarında. Onnara verilmiş akıl hem üz da läazım biri-birinä davransınnar kardaşlık ruhuna uygun.
Turkmen Adamlaryň hemmesi azat dogulýarlar we öz mertebesi hem-de hukuklary boýunça ilkibaşdan deňdirler. Olara ozal-başdan aň, ynsap berlendir we biri-birine özara doganlyk ruhunda çemeleşmek olaryň ýaraşygydyr.
Uzbek (Latin script) Barcha odamlar erkin, qadr-qimmat ba huquqlarda tang bo'lib tug'iladilar. Ular aql va vijdon sohibidilar va bir-birla iga birodarlarcha muomala qilishlari zarur.
Uyghur ھەممە ئادەم زاتىدىنلا ئەركىن، ئىززەت-ھۆرمەت ۋە ھوقۇقتا بابباراۋەر بولۇپ تۇغۇلغان. ئۇلار ئەقىلغە ۋە ۋىجدانغا ئىگە ھەمدە بىر-بىرىگە قېرىنداشلىق مۇناسىۋىتىگە خاس روھ بىلەن مۇئامىلە قىلىشى كېرەك.
Tyva Бүгү кижилер хостуг база мөзүзү болгаш эргелери дең кылдыр төрүттүнер. Оларга угаансарыыл болгаш арын-нүүр бердинген болур болгаш олар бот-боттарынга акы-дуңмалышкы хамаарылганы көргүзер ужурлуг.
Karachay-Balkar Бютеу адамла эркин болуб эмда сыйлары бла хакълары тенг болуб тууадыла. Алагъа акъыл бла намыс берилгенди эмда бир-бирлерине къарнашлыкъ халда къараргъа керекдиле.
Altai Ончо улус ак-јарыкка јайым ла теҥ-тай тап-эриктӱ туулат. Олор санааукаалу ла чек кӱӱн-тапту болуп бӱткен ле бой-бойын карындаш кирези кӧрӧр лӧ јӱрер учурлу.
Khakas Полған на кізі пос паза тиң тіріпче паза тиң постың синін пілінгенін паза тірелерініңде полча. Олардың сағынғаны паза арығ сағыс пар паза харындастар чіли тудынарға киректер.
Shor Парчын кижи, по чарыққа туғчадып, тең, пош туғча. Кижилер сағыштығ, ақтығ туғчалар, кижилерге пашқа кижилербе арғыштаныштарға керек.
Sakha Дьон барыта бэйэ суолтатыгар уонна быраабыгар тэҥ буолан төрүүллэр. Кинилэр бары өркөн өйдөөх, суобастаах буолан төрүүллэр, уонна бэйэ бэйэлэригэр тылга кииринигэс быһыылара доҕордоһуу тыыннаах буолуохтаах.
Crimean Tatar Bütün insanlar ür, sayğı ve uquqlar itibarı ile musaviy doğalar. Aqıl ve vicdanğa saipler ve bir-birlerine qarşı ağa-qardaşlıq fikirnen areket etmelidirler.

Vulnerability

This table summarizes the vulnerability of various Turkic languages. Vulnerability data is derived from the ‘Atlas of the World’s Languages in Danger, © UNESCO, http://www.unesco.org/culture/languages-atlas’ and Ethnologue.

Language ISO639-3 Location Speakers Status
Ethnologue UNESCO
Chagatai chg Turkmenistan 0 10 (Extinct) -
Salchuq slq Iran 0 10 (Extinct) -
Chulym clw Russian Federation 44 8b (Nearly extinct) 4 (Critically endangered)
Karaim kdr Lithuania & Ukraine 56 8b (Nearly extinct) 5 (Extinct)
Tofa (Karagas) kim Russian Federation 90 8b (Nearly extinct) 4 (Critically endangered)
Krimchak jct Ukraine 200 8b (Nearly extinct) -
Ili Turki ili China 120 8a (Moribund) 3 (Severely endangered)
Yugur (Western/Saryg) ybe China 4,600 8a (Moribund) 3 (Severely endangered)
Balkan Gagauz Turkish bgx Turkey 331,000 7 (Shifting) -
Dolgan dlg Russian Federation 1,050 6b (Threatened) 2 (Definitely endangered)
Shor cjs Russian Federation 2,840 6b (Threatened) 3 (Severely endangered)
Ainu aib China 6,570 6b (Threatened) -
Altay, Northern atv Russian Federation 57,400 6b (Threatened) 3 (Severely endangered)
Salar slr China 70,000 6b (Threatened) 1 (Vulnerable)
Urum uum Georgia, Russian Federation, Ukraine 192,700 6b (Threatened) 2 (Definitely endangered)
Khalaj klj Iran 42,100 6a (Vigorous) 1 (Vulnerable)
Khorasani kmz Iran 400,000 6a (Vigorous) 1 (Vulnerable)
Kashkay qxq Iran 1,500,000 6a (Vigorous) -
Khakas kjh Russian Federation 42,610 5 (Developing) 5 (Extinct)
Nogay nog Russian Federation 87,410 5 (Developing) 2 (Definitely endangered)
Gagauz gag Moldova 173,920 5 (Developing) -
Karakalpak kaa Uzbekistan 410,410 5 (Developing) -
Kumyk kum Russian Federation 426,550 5 (Developing) 1 (Vulnerable)
Tuva tyv China, Mongolia, Russian Federation 283,400 4 (Educational) 1 (Vulnerable)
Karachay-Balkar krc Russian Federation 310,730 4 (Educational) 1 (Vulnerable)
Crimean Tatar crh Ukraine 475,540 4 (Educational) -
Chuvash chv Russian Federation 1,077,420 4 (Educational) 1 (Vulnerable)
Bashkort bak Russian Federation 1,221,340 4 (Educational) 1 (Vulnerable)
Uzbek, Southern uzs Afghanistan 2,961,980 3 (Wider communication) -
Azerbaijani, Southern azb Iran 16,913,280 3 (Wider communication) -
Altay, Southern alt Russian Federation 57,400 2 (Provincial) 2 (Definitely endangered)
Yakut sah Russian Federation 450,000 2 (Provincial) 1 (Vulnerable)
Tatar tat Russian Federation 5,407,550 2 (Provincial) 2 (Definitely endangered)
Uyghur uig China 8,791,610 2 (Provincial) -
Kyrgyz kir Kyrgyzstan 2,941,930 1 (National) -
Azerbaijani, Northern azj Azerbaijan 7,324,270 1 (National) -
Turkmen tuk Turkmenistan 7,560,560 1 (National) -
Kazakh kaz Kazakhstan 8,077,770 1 (National) -
Uzbek, Northern uzn Uzbekistan 18,968,250 1 (National) -
Turkish tur Turkey 50,733,420 1 (National) -

Official poem

  • (tur) Kovayla bira içerim, ama sen bilmezsin. Yarın gelir misin?
  • (aze) Vedrəyle pivə içirəm, ama sen bilməzsən. Yarın gələrmisən?
  • (tat) Чиләк белән сыра эчәм, әмма син белмисең. Иртәгә киләсеңме?
  • (kaz) Шелекпен сыра ішемін, бірақ сен білмейсің. Ертең келесің бе?
  • (kir) Чака менен сыра ичем, бирок сен билбейсиң. Эртең келесиңби?
  • (uzb) Chelak bilan pivo ichaman lekin sen bilmaysan. Ertaga kelasanmi?
  • (kaa) Shelek penen pivo ishemen, biraq sen bilmeysen'. Erten' kelesen'be?
  • (tlt) Кӧнӧкмынаң сыра ичип јадым, сен-оқ аның керегинде пилбей јазың. Таңда келерзиң бе?

Footnotes


See also

Bibliography

  • Sundetova, A., Tyers, F. M. and Forcada, M. L. (2015) "A free/open-source machine translation system for English to Kazakh". Proceedings of the 3rd Conference on Turkic Languages (TurkLang 2015).
  • Tyers, F. M. and Washington, J. (2015) "Towards a free/open-source universal-dependency treebank for Kazakh". Proceedings of the 3rd Conference on Turkic Languages (TurkLang 2015).
  • Washington, J. N., Salimzyanov, I., and Tyers, F. M. (2014) "Finite-state morphological transducers for three Kypchak languages". Proceedings of the 9th Conference on Language Resources and Evaluation, LREC2014
  • Salimzyanov, I., Washington, J. N., and Tyers, F. M. (2013) "A free/open-source Kazakh-Tatar machine translation system". Proceedings of MT Summit XIV
  • Washington, J. N., Ipasov, M. and Tyers, F. M. (2012) "A finite-state morphological analyser for Kyrgyz". Proceedings of the 8th Conference on Language Resources and Evaluation, LREC2012
  • Tyers, F. M., Washington, J. N., Salimzyanov, I., and Batalov, R. (2012) "A prototype machine translation system for Tatar and Bashkir based on free/open-source components". Proceedings of the Turkic Languages Workshop at the Language Resources and Evaluation Conference, LREC2012
Retrieved from "https://wiki.apertium.org/w/index.php?title=Turkic_languages&oldid=73280"