Difference between revisions of "Indic languages"

From Apertium
Jump to navigation Jump to search
(30 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
{{TOCD}}
 
{{TOCD}}
  +
The '''Indic languages''' include [[Hindi]], [[Urdu]], [[Bengali]], [[Sanskrit]], and a number of other languages. These languages are the dominant language family of the Indian subcontinent. The number of people that speak an Indic language is upwards of 900,000,000.
===IN PROGRESS===
 
 
The '''Indic languages''' include [[Hindi]], [[Urdu]], [[Bengali]], [[Sanskrit]], and several other languages. These languages are the dominant language family of the Indian subcontinent. The number of people that speak an Indic language is upwards of 900,000,000.
 
   
 
The master plan involves generating independent finite-state transducers for each language, and then making individual dictionaries and transfer rules for every pair. The current status of these goals is listed below.
 
The master plan involves generating independent finite-state transducers for each language, and then making individual dictionaries and transfer rules for every pair. The current status of these goals is listed below.
Line 20: Line 18:
 
!rowspan=2| state
 
!rowspan=2| state
 
!rowspan=2| stems
 
!rowspan=2| stems
  +
!rowspan=2| paradigms
 
!rowspan=2| coverage
 
!rowspan=2| coverage
 
!rowspan=2| location
 
!rowspan=2| location
Line 32: Line 31:
 
|| <code>san</code>
 
|| <code>san</code>
 
|| [[lttoolbox]]
 
|| [[lttoolbox]]
|| production
+
|| working
 
|align="right"| {{#lst:Apertium-san/stats|stems}}
 
|align="right"| {{#lst:Apertium-san/stats|stems}}
  +
|align="right"| {{#lst:Apertium-san/stats|paradigms}}
 
|align="center"| -
 
|align="center"| -
 
|| [[apertium-san]] ([[languages]])
 
|| [[apertium-san]] ([[languages]])
 
|| Amba Kulkarni
 
|| Amba Kulkarni
 
|-
 
|-
  +
|| <code>[[apertium-urd]]</code>
 
|| [[Urdu]]
 
|| <code>ur</code>
 
|| <code>urd</code>
 
|| [[lttoolbox]]
 
|| [[nursery]]
 
|align="right"| {{#lst:apertium-urd/stats|stems}}
 
|align="center"| -
 
|| [[apertium-urd]]&nbsp;([[languages]])
 
|| Muhammad Humayoun
 
|-
 
 
|| <code>[[apertium-hin]]</code>
 
|| <code>[[apertium-hin]]</code>
 
|| [[Hindi]]
 
|| [[Hindi]]
Line 54: Line 44:
 
|| <code>hin</code>
 
|| <code>hin</code>
 
|| [[lttoolbox]]
 
|| [[lttoolbox]]
|| production
+
|| working
 
|align="right"| {{#lst:apertium-hin/stats|stems}}
 
|align="right"| {{#lst:apertium-hin/stats|stems}}
|align="center"| -
+
|align="right"| {{#lst:apertium-hin/stats|paradigms}}
  +
|align="center"| ~{{#lst:apertium-hin/stats/average}}%
 
|| [[apertium-hin]]&nbsp;([[languages]])
 
|| [[apertium-hin]]&nbsp;([[languages]])
 
|| [[User:Nikant|Nikant]], [[User:darthxaher|Abu Zaher Md. Faridee]], [[User:Francis Tyers|Fran]]
 
|| [[User:Nikant|Nikant]], [[User:darthxaher|Abu Zaher Md. Faridee]], [[User:Francis Tyers|Fran]]
Line 65: Line 56:
 
|| <code>ben</code>
 
|| <code>ben</code>
 
|| [[lttoolbox]]
 
|| [[lttoolbox]]
|| [[nursery]]
+
|| development
 
|align="right"| {{#lst:apertium-ben/stats|stems}}
 
|align="right"| {{#lst:apertium-ben/stats|stems}}
|align="center"| -
+
|align="right"| {{#lst:apertium-ben/stats|paradigms}}
  +
|align="center"| ~{{#lst:apertium-ben/stats/average}}%
 
|| [[apertium-ben]]&nbsp;([[languages]])
 
|| [[apertium-ben]]&nbsp;([[languages]])
 
|| [[User:darthxaher|Abu Zaher Md. Faridee]]
 
|| [[User:darthxaher|Abu Zaher Md. Faridee]]
 
|-
 
|-
  +
|| <code>[[apertium-urd]]</code>
  +
|| [[Urdu]]
  +
|| <code>ur</code>
  +
|| <code>urd</code>
  +
|| [[lttoolbox]]
  +
|| development
  +
|align="right"| {{#lst:apertium-urd/stats|stems}}
  +
|align="right"| {{#lst:apertium-urd/stats|paradigms}}
  +
|align="center"| ~{{#lst:apertium-urd/stats/average}}%
  +
|| [[apertium-urd]]&nbsp;([[languages]])
  +
|| Muhammad Humayoun
  +
|-
  +
|| <code>[[apertium-nep]]</code>
  +
|| [[Nepali]]
  +
|| <code>ne</code>
  +
|| <code>nep</code>
  +
|| [[lttoolbox]]
  +
|| prototype
  +
|align="right"| {{#lst:apertium-nep/stats|stems}}
  +
|align="right"| {{#lst:apertium-nep/stats|paradigms}}
  +
|align="center"| -
  +
|| [[apertium-ne-en]]&nbsp;([[incubator]])<br />[[apertium-eo-ne]]&nbsp;([[incubator]])
  +
||
  +
|-
  +
|| <code>[[apertium-mar]]</code>
  +
|| [[Marathi]]
  +
|| <code>mr</code>
  +
|| <code>mar</code>
  +
|| [[lttoolbox]]
  +
|| prototype
  +
|align="right"| {{#lst:apertium-mar/stats|stems}}
  +
|align="right"| {{#lst:apertium-mar/stats|paradigms}}
  +
|align="center"| -
  +
|| [[apertium-mar-eng]]&nbsp;([[incubator]])<br />[[apertium-mr-hi]]&nbsp;([[incubator]])
  +
||
  +
|-
  +
|| <code>[[apertium-sin]]</code>
  +
|| [[Sinhala]]
  +
|| <code>si</code>
  +
|| <code>sin</code>
  +
|| [[lttoolbox]]
  +
|| {{#lst:apertium-sin/stats|state}}
  +
|align="right"| {{#lst:apertium-sin/stats|stems}}
  +
|align="right"| {{#lst:apertium-sin/stats|paradigms}}
  +
|align="center"| -
  +
|| {{#lst:apertium-sin/stats|location}}
  +
|| {{#lst:apertium-sin/stats|authors}}
  +
|-
  +
|-
  +
|| <code>[[apertium-pan]]</code>
  +
|| [[Punjabi]]
  +
|| <code>pa</code>
  +
|| <code>pan</code>
  +
|| —
  +
|| possibly non-existant
  +
|align="right"| -
  +
|align="right"| -
  +
|align="center"| -
  +
|| [[apertium-pa-hi]]&nbsp;([[incubator]])<br />[[apertium-ur-pa]]&nbsp;([[incubator]])
  +
||
 
|}
 
|}
 
   
 
=== Indic Language Classification ===
 
=== Indic Language Classification ===
Line 86: Line 137:
 
* [[Sanskrit]]
 
* [[Sanskrit]]
 
* Southern Zone: [[Marathi]], [[Konkani]], [[Urdu]]
 
* Southern Zone: [[Marathi]], [[Konkani]], [[Urdu]]
** Insular Indic: [[Sinhalese]], [[Maldivian]]
+
** Insular Indic: [[Sinhala]], [[Maldivian]]
 
* Eastern Zone: [[Bengali]], [[Oriya]], [[Tharu]]
 
* Eastern Zone: [[Bengali]], [[Oriya]], [[Tharu]]
 
** Bihari: [[Bhojpuri]], [[Maithili]], etc.
 
** Bihari: [[Bhojpuri]], [[Maithili]], etc.
 
   
 
=== Existing language pairs ===
 
=== Existing language pairs ===
  +
Text in ''italics'' denotes language pairs in the incubator. Regular text denotes a developing language pair in nursery, while text in '''bold''' denotes a stable well-working language pair in trunk and text in '''''bold and italics''''' denotes a pair in staging. Bidix stems as counted with [[dixcounter]] are displayed below.
   
  +
{| style="text-align: center;" class="wikitable dixtable"
==== Indic-Indic pairs ====
 
 
 
{| style="text-align: center;" class="wikitable"
 
 
|- style="background: #ececec"
 
|- style="background: #ececec"
! !! hin !! ben !! urd !! san !! nep !! mar !! pan
+
! !! hin !! ben !! urd !! san !! nep !! mar !! pan !! sin
 
|-
 
|-
  +
| '''hin''' || - || ''[[Apertium-bn-hi|bn-hi]]''<br>{{#lst:Apertium-bn-hi/stats|bn-hi_stems}} || '''[[Apertium-urd-hin|urd-hin]]'''<br>'''{{#lst:Apertium-urd-hin/stats|urd-hin_stems}}''' || || || ''[[Apertium-mar-hin|mar-hin]]''<br>{{#lst:Apertium-mar-hin/stats|mar-hin_stems}} || ''[[Apertium-pa-hi|pa-hi]]''<br>{{#lst:Apertium-pa-hi/stats|pa-hi_stems}} ||
| '''hin''' || - || || || || || ||
 
 
|-
 
|-
| '''ben''' || [[bn-hi]] || - || || || || ||
+
| '''ben''' || ''[[Apertium-bn-hi|bn-hi]]''<br>{{#lst:Apertium-bn-hi/stats|bn-hi_stems}} || - || || || || || ||
 
|-
 
|-
| '''urd''' || [[ur-hi]] || || - || || || || [[ur-pa]]
+
| '''urd''' || '''[[Apertium-urd-hin|urd-hin]]'''<br>'''{{#lst:Apertium-urd-hin/stats|urd-hin_stems}}''' || || - || || || || ''[[Apertium-ur-pa|ur-pa]]''<br>{{#lst:Apertium-ur-pa/stats|ur-pa_stems}} ||
 
|-
 
|-
| '''san''' || || || || - || || ||
+
| '''san''' || || || || - || || || ||
 
|-
 
|-
| '''nep''' || || || || || - || ||
+
| '''nep''' || || || || || - || || ||
 
|-
 
|-
| '''mar''' || [[mr-hi]] || || || || || - ||
+
| '''mar''' || ''[[Apertium-mar-hin|mar-hin]]''<br>{{#lst:Apertium-mar-hin/stats|mar-hin_stems}} || || || || || - || ||
 
|-
 
|-
| '''pan''' || [[pa-hi]] || || || || || || -
+
| '''pan''' || ''[[Apertium-pa-hi|pa-hi]]''<br>{{#lst:Apertium-pa-hi/stats|pa-hi_stems}} || || ''[[Apertium-ur-pa|ur-pa]]''<br>{{#lst:Apertium-ur-pa/stats|ur-pa_stems}} || || || || - ||
  +
|-
  +
| '''sin''' || || || || || || || || -
  +
|-
  +
| || || || || || || || ||
  +
|-
  +
| '''asm''' || ''[[Apertium-as-hi|as-hi]]''<br>{{#lst:Apertium-as-hi/stats|as-hi_stems}} || ''[[Apertium-asm-ben|asm-ben]]''<br>{{#lst:Apertium-asm-ben/stats|asm-ben_stems}} || || || || || ||
  +
|-
  +
| '''eng''' || [[Apertium-eng-hin|eng-hin]]<br>{{#lst:Apertium-eng-hin/stats|eng-hin_stems}} || ''[[Apertium-bn-en|bn-en]]''<br>{{#lst:Apertium-bn-en/stats|bn-en_stems}} || || || ''[[Apertium-ne-en|ne-en]]''<br>{{#lst:Apertium-ne-en/stats|ne-en_stems}} || ''[[Apertium-mar-eng|mar-eng]]''<br>{{#lst:Apertium-mar-eng/stats|mar-eng_stems}} || || ''[[Apertium-si-en|si-en]]''<br>{{#lst:Apertium-si-en/stats|si-en_stems}}
  +
|-
  +
| '''epo''' || || || || || ''[[Apertium-eo-ne|eo-ne]]''<br>{{#lst:Apertium-eo-ne/stats|eo-ne_stems}} || || ||
  +
|-
  +
| '''fas''' || || || ''[[Apertium-ur-fa|ur-fa]]''<br>{{#lst:Apertium-ur-fa/stats|ur-fa_stems}} || || || || ||
 
|}
 
|}
   
  +
==Samples==
==== Pairs with non-Indic languages ====
 
  +
Article 1 of the Universal Declaration of Human Rights:
{| style="text-align: center;" class="wikitable"
 
  +
|- style="background: #ececec"
 
  +
''All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.''
! !! eng !! asm !! epo !! fas
 
  +
  +
{|class=wikitable
  +
! Language !! Text
 
|-
 
|-
  +
|| Sanskrit || सर्वे मानवा: स्वतन्त्रा: समुत्पन्ना: वर्तन्ते अपि च, गौरवदृशा अधिकारदृशा च समाना: एव वर्तन्ते। एते सर्वे चेतना-तर्क-शक्तिभ्यां सुसम्पन्ना: सन्ति। अपि च, सर्वेऽपि बन्धुत्व-भावनया परस्परं व्यवहरन्तु।
| '''hin''' || [[eng-hin]] || [[as-hi]] || [[mr-hi]] ||
 
 
|-
 
|-
  +
|| Hindi || सभी मनुष्यों को गौरव और अधिकारों के मामले में जन्मजात स्वतन्त्रता और समानता प्राप्त है । उन्हें बुद्घि और अन्तरात्मा की देन प्राप्त है और परस्पर उन्हें भाईचारे के भाव से बर्ताव करना चाहिए ।
| '''ben''' || [[bn-en]] || || ||
 
 
|-
 
|-
  +
|| Bengali || সমস্ত মানুষ স্বাধীনভাবে সমান মর্যাদা এবং অধিকার নিয়ে জন্মগ্রহণ করে। তাঁদের বিবেক এবং বুদ্ধি আছে ; সুতরাং সকলেরই একে অপরের প্রতি ভ্রাতৃত্বসুলভ মনোভাব নিয়ে আচরণ করা উচিত্‍।
| '''urd''' || || || || [[ur-fa]]
 
 
|-
 
|-
  +
|| Urdu || تمام انسان آزادی اور حقوق و عزت کے اعتبار سے برابر پیدا ہویٔے ہیں۔ انہیں ضمیر اور عقل و دیعت ہویٔی ہے۔ اس لیٔے انہیں ایک دوسرے کے ساتھ بھایٔی چارے کا سلوک کرنا چاہیء۔
| '''san''' || || || ||
 
 
|-
 
|-
  +
|| Nepali || सबै व्यक्ति हरू जन्मजात स्वतन्त्र हुन ती सबैको समान अधिकार र महत्व छ। निजहरूमा विचार शक्ति र सद्धिचार भएकोले निजहरूले आपसमा भातृत्वको भावना बाट व्यवहार गर्नु पर्छ।
| '''nep''' || [[ne-en]] || || [[eo-ne]] ||
 
 
|-
 
|-
  +
|| Marathi || सर्व मानवी व्यक्ति जन्मतःच स्वतंत्र आहेत व त्यांना समान प्रतिष्ठा व समान अधिकार आहेत. त्यांना विचारशक्ति व सदसविद्वेकबुद्धि लाभलेली आहे. व त्यांनी एकमेकांशी बंधुत्याच्या भावनेने आचरण करावे.
| '''mar''' || [[mar-eng]] || || ||
 
 
|-
 
|-
  +
|| Panjabi, Eastern || ਸਾਰਾ ਮਨੁੱਖੀ ਪਰਿਵਾਰ ਆਪਣੀ ਮਹਿਮਾ, ਸ਼ਾਨ ਅਤੇ ਹੱਕਾਂ ਦੇ ਪੱਖੋਂ ਜਨਮ ਤੋਂ ਹੀ ਆਜ਼ਾਦ ਹੈ ਅਤੇ ਸੁਤੇ ਸਿੱਧ ਸਾਰੇ ਲੋਕ ਬਰਾਬਰ ਹਨ । ਉਨ੍ਹਾਂ ਸਭਨਾ ਨੂੰ ਤਰਕ ਅਤੇ ਜ਼ਮੀਰ ਦੀ ਸੌਗਾਤ ਮਿਲੀ ਹੋਈ ਹੈ ਅਤੇ ਉਨ੍ਹਾਂ ਨੂੰ ਭਰਾਤਰੀਭਾਵ ਦੀ ਭਾਵਨਾ ਰਖਦਿਆਂ ਆਪਸ ਵਿਚ ਵਿਚਰਣਾ ਚਾਹੀਦਾ ਹੈ ।
| '''pan''' || || || ||
 
 
|}
 
|}
   
 
==Tagset==
 
==Tagset==
   
Rough guide to tagsets in various Indic language transducers, with an eye to keeping stuff that is basically the same tagged the same. In the following table, <sup>A</sup> stands for Apertium and <sup>T</sup> stands for [[TRmorph]] (See also [[List_of_symbols|the general tagset list]]).
+
Rough guide to tagsets in various Indic language transducers, with an eye to keeping stuff that is basically the same tagged the same (see also [[List_of_symbols|the general tagset list]]).
   
 
{|class="wikitable"
 
{|class="wikitable"
Line 145: Line 208:
 
|-
 
|-
 
| Noun || || || {{tag|n}} || ||
 
| Noun || || || {{tag|n}} || ||
  +
|}
  +
  +
[[Category:Indic languages]]

Revision as of 21:49, 3 September 2017

The Indic languages include Hindi, Urdu, Bengali, Sanskrit, and a number of other languages. These languages are the dominant language family of the Indian subcontinent. The number of people that speak an Indic language is upwards of 900,000,000.

The master plan involves generating independent finite-state transducers for each language, and then making individual dictionaries and transfer rules for every pair. The current status of these goals is listed below.

Status

The ultimate goal is to have multi-purposable transducers for a variety of Indic languages. These can then be paired for X→Y translation with the addition of a CG for language X and transfer rules / dictionary for the pair X→Y. Below is listed development progress for each language's transducers and dictionary pairs.

Transducers

Once a transducer has ~80% coverage on a range of medium-large corpora we can say it is "working". Over 90% and it can be considered to be "production".

name Language ISO 639 formalism state stems paradigms coverage location primary authors
-2 -3
apertium-san Sanskrit sa san lttoolbox working 123,373 279 - apertium-san (languages) Amba Kulkarni
apertium-hin Hindi hi hin lttoolbox working 37,833 101 ~83.1% apertium-hin (languages) Nikant, Abu Zaher Md. Faridee, Fran
apertium-ben Bengali bn ben lttoolbox development 8,230 137 ~74% apertium-ben (languages) Abu Zaher Md. Faridee
apertium-urd Urdu ur urd lttoolbox development 14,943 123 ~64.6% apertium-urd (languages) Muhammad Humayoun
apertium-nep Nepali ne nep lttoolbox prototype 215 - apertium-ne-en (incubator)
apertium-eo-ne (incubator)
apertium-mar Marathi mr mar lttoolbox prototype 14,886 111 - apertium-mar-eng (incubator)
apertium-mr-hi (incubator)
apertium-sin Sinhala si sin lttoolbox prototype 73 22 - apertium-si-en (incubator)
apertium-pan Punjabi pa pan possibly non-existant - - - apertium-pa-hi (incubator)
apertium-ur-pa (incubator)

Indic Language Classification

Existing language pairs

Text in italics denotes language pairs in the incubator. Regular text denotes a developing language pair in nursery, while text in bold denotes a stable well-working language pair in trunk and text in bold and italics denotes a pair in staging. Bidix stems as counted with dixcounter are displayed below.

hin ben urd san nep mar pan sin
hin - bn-hi
219
urd-hin
4,943
mar-hin
15,857
pa-hi
51,577
ben bn-hi
219
-
urd urd-hin
4,943
- ur-pa
0
san -
nep -
mar mar-hin
15,857
-
pan pa-hi
51,577
ur-pa
0
-
sin -
asm as-hi
833
asm-ben
969
eng eng-hin
39,242
bn-en
7,497
ne-en
163
mar-eng
102
si-en
80
epo eo-ne
9,306
fas ur-fa
?

Samples

Article 1 of the Universal Declaration of Human Rights:

All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.

Language Text
Sanskrit सर्वे मानवा: स्वतन्त्रा: समुत्पन्ना: वर्तन्ते अपि च, गौरवदृशा अधिकारदृशा च समाना: एव वर्तन्ते। एते सर्वे चेतना-तर्क-शक्तिभ्यां सुसम्पन्ना: सन्ति। अपि च, सर्वेऽपि बन्धुत्व-भावनया परस्परं व्यवहरन्तु।
Hindi सभी मनुष्यों को गौरव और अधिकारों के मामले में जन्मजात स्वतन्त्रता और समानता प्राप्त है । उन्हें बुद्घि और अन्तरात्मा की देन प्राप्त है और परस्पर उन्हें भाईचारे के भाव से बर्ताव करना चाहिए ।
Bengali সমস্ত মানুষ স্বাধীনভাবে সমান মর্যাদা এবং অধিকার নিয়ে জন্মগ্রহণ করে। তাঁদের বিবেক এবং বুদ্ধি আছে ; সুতরাং সকলেরই একে অপরের প্রতি ভ্রাতৃত্বসুলভ মনোভাব নিয়ে আচরণ করা উচিত্‍।
Urdu تمام انسان آزادی اور حقوق و عزت کے اعتبار سے برابر پیدا ہویٔے ہیں۔ انہیں ضمیر اور عقل و دیعت ہویٔی ہے۔ اس لیٔے انہیں ایک دوسرے کے ساتھ بھایٔی چارے کا سلوک کرنا چاہیء۔
Nepali सबै व्यक्ति हरू जन्मजात स्वतन्त्र हुन ती सबैको समान अधिकार र महत्व छ। निजहरूमा विचार शक्ति र सद्धिचार भएकोले निजहरूले आपसमा भातृत्वको भावना बाट व्यवहार गर्नु पर्छ।
Marathi सर्व मानवी व्यक्ति जन्मतःच स्वतंत्र आहेत व त्यांना समान प्रतिष्ठा व समान अधिकार आहेत. त्यांना विचारशक्ति व सदसविद्वेकबुद्धि लाभलेली आहे. व त्यांनी एकमेकांशी बंधुत्याच्या भावनेने आचरण करावे.
Panjabi, Eastern ਸਾਰਾ ਮਨੁੱਖੀ ਪਰਿਵਾਰ ਆਪਣੀ ਮਹਿਮਾ, ਸ਼ਾਨ ਅਤੇ ਹੱਕਾਂ ਦੇ ਪੱਖੋਂ ਜਨਮ ਤੋਂ ਹੀ ਆਜ਼ਾਦ ਹੈ ਅਤੇ ਸੁਤੇ ਸਿੱਧ ਸਾਰੇ ਲੋਕ ਬਰਾਬਰ ਹਨ । ਉਨ੍ਹਾਂ ਸਭਨਾ ਨੂੰ ਤਰਕ ਅਤੇ ਜ਼ਮੀਰ ਦੀ ਸੌਗਾਤ ਮਿਲੀ ਹੋਈ ਹੈ ਅਤੇ ਉਨ੍ਹਾਂ ਨੂੰ ਭਰਾਤਰੀਭਾਵ ਦੀ ਭਾਵਨਾ ਰਖਦਿਆਂ ਆਪਸ ਵਿਚ ਵਿਚਰਣਾ ਚਾਹੀਦਾ ਹੈ ।

Tagset

Rough guide to tagsets in various Indic language transducers, with an eye to keeping stuff that is basically the same tagged the same (see also the general tagset list).

Phenomenon Morphology Description Tag(s) Language(s) Notes
Part of speech
Noun <n>