User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress - Revision history

Pmodi: /* Repos */

2020-09-07T10:49:26Z

Repos

Pmodi: /* Dictionaries */

2020-08-31T07:51:02Z

Dictionaries

Pmodi: /* Repos */

2020-08-31T07:45:27Z

Repos

Pmodi: /* Repos */

2020-08-31T07:39:59Z

Repos

Pmodi at 16:00, 30 August 2020

2020-08-30T16:00:18Z

Pmodi: /* Repos */

2020-08-30T15:53:50Z

Repos

Pmodi: /* Repos */

2020-08-30T15:53:33Z

Repos

Pmodi: /* Future Work */

2020-08-30T15:52:23Z

Future Work

Pmodi at 15:50, 30 August 2020

2020-08-30T15:50:13Z

Pmodi: /* Progress table */

2020-08-29T21:39:33Z

Progress table

@@ Line 8: / Line 8: @@
 Most code was finally merged and can be found here : https://apertium.projectjj.com/gsoc2020/priyankmodiPM.html . Thanks to Tino :)
-PRs are still pending to be reviewed :
-* https://github.com/apertium/apertium-hin/pull/11
-* https://github.com/apertium/apertium-pan/pull/14
-* https://github.com/apertium/apertium-pan/pull/13
 ==Main Work==

@@ Line 19: / Line 19: @@
 The major accomplishment of this project has been in the Punjabi monodix which reached 85% Coverage(with over 12,000 stems) on a total of 3.6 million tokens(entire Wikipedia corpus on Punjabi). In addition to new entries (added from frequency lists and crossdics), entries broken due to tag changes have been fixed, and the dictionary has been reorganized and cleaned. Additions have also been made to the Hindi monodix, also changing paradigm in cases handled by separable.
-The bidix also has close to 17,500 translations and the pan-hin translator coverage is also just above 80% computed on the same corpus. Weights have been assigned to the entries as well because of multiple spelling options for a translation. This will be taken up in future work to automatically assign the correct spelling.
+The bidix also has close to 17,500 translations and the pan-hin translator coverage is also just above 80% computed on the same corpus. Weights have been assigned to the entries as well because of multiple spelling options for a translation. This will be taken up in future work to automatically assign the correct spelling. The hin-pan translator also has decent coverage(~100% for post/prepositions, pronouns, proper nouns, adverbs, adjectives, intransitive verbs. >80% in case of adjectives. ~90% in case of transitive verbs. Most nouns high up/midway on the zipf curve have been added, but there's still quite a few with lower frequencies that remain to be added).
 The pair still needs to go through testvoc iterations as there exist some verbs which existed in the bidix before the project. Most of these have been removed and Proper nouns, adverbs, adjectives and almost all nouns have been checked.

@@ Line 8: / Line 8: @@
 Most code was finally merged and can be found here : https://apertium.projectjj.com/gsoc2020/priyankmodiPM.html . Thanks to Tino :)
 PRs are still pending to be reviewed :
 * https://github.com/apertium/apertium-hin/pull/11

@@ Line 7: / Line 7: @@
 * https://github.com/apertium/apertium-hin
-All code was finally merged and can be found here : https://apertium.projectjj.com/gsoc2020/priyankmodiPM.html . Thanks to Tino :)
+Most code was finally merged and can be found here : https://apertium.projectjj.com/gsoc2020/priyankmodiPM.html . Thanks to Tino :)
+PRs are still pending to be reviewed :
+* https://github.com/apertium/apertium-hin/pull/11
+* https://github.com/apertium/apertium-pan/pull/14
+* https://github.com/apertium/apertium-pan/pull/13
 ==Main Work==

@@ Line 296: / Line 296: @@
 * Lexical selection rules need to be added. Could not be completed due to some errors in IRSTLM. For example, a simple rule : aisa <noun> - ajiha BUT aisa <verb> innj.
 * Meeting testvoc requirements before release.
+==Experience and Final words==
+Working with Apertium over the past three months has been nothing less than amazing. I don't think I ever had a problem understanding the pipe(the limited parts of it I used) because of the extensive documentation that exists. The community is probably the most helpful one I've ever been part of. Would love to see this pair out soon, after ofcourse a good analysis of my work.
 ==Literature(Apertium Wiki) Covered==

@@ Line 289: / Line 289: @@
 * Multiple paradigms, mostly in category <prn> use the "mf" in place of m-obj. This needs to be fixed in multiple places in the hindi, urdu dictionaries. For example : Number of subject-object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ. मेरे(marked mf) | मेरा(marked m)
 * Tagger needs to be trained to pick verbs and post/pre positions preferred to adjectives, adverbs(this is not a regular trend and results are mostly decent)
-* In some slangs, postpositions are combined with words. Need to identify and these. For examples : Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ.
+* In some slangs, post positions are combined with words. Need to identify these. For examples : Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ. It's a very irregular trend though and usually not used in written Punjabi.
-* What is the analysis for ਦੋਸਤੋਂ ("Friends, come with me). The problem will be that doston in Hindi is the translation for this and friends being used as in (his friends did this). Not a big problem, but needs analysis to idenify more such occurences.
+* What is the analysis for ਦੋਸਤੋਂ ("Friends, come with me). The problem will be that doston in Hindi is the translation for this and friends being used as in (his friends did this). Not a big problem, but needs analysis to identify more such occurrences.
 * Add transfer rule and check if tagger correctly identifies which one is being picked. ਮੈਂ is the translation for मैंने and मैं.
 * Fix alternate spellings analyses. Mostly multiple spellings have been added but there could be some easy automatic fixes which can be done. For example : ਉਸ੍ਸਰ > ਉੱਸਰ, ਖਿਲ੍ਲਰ > ਖਿੱਲਰ
+* Fix for borrowed words(mostly English) and Proper nouns. Can work on transliteration once these NEs can be correctly identified.
-* कि v/s की punjabi ki can also be kya.
-* Fix for borrowed words(mostly english) and Proper nouns. Can work on transliteration once these NEs can be correctly identified.
 * Lexical selection rules need to be added. Could not be completed due to some errors in IRSTLM. For example, a simple rule : aisa <noun> - ajiha BUT aisa <verb> innj.
+* Meeting testvoc requirements before release.
 ==Literature(Apertium Wiki) Covered==

@@ Line 8: / Line 8: @@
 All code was finally merged.
+==Main Work==
+===Dictionaries===
+The major accomplishment of this project has been in the Punjabi monodix which reached 85% Coverage(with over 12,000 stems) on a total of 3.6 million tokens(entire Wikipedia corpus on Punjabi). In addition to new entries (added from frequency lists and crossdics), entries broken due to tag changes have been fixed, and the dictionary has been reorganized and cleaned. Additions have also been made to the Hindi monodix, also changing paradigm in cases handled by separable.
+The bidix also has close to 17,500 translations and the pan-hin translator coverage is also just above 80% computed on the same corpus. Weights have been assigned to the entries as well because of multiple spelling options for a translation. This will be taken up in future work to automatically assign the correct spelling.
+The pair still needs to go through testvoc iterations as there exist some verbs which existed in the bidix before the project. Most of these have been removed and Proper nouns, adverbs, adjectives and almost all nouns have been checked.
+===Separable===
+Thanks to the Lsx module, Apertium can efficiently handle multiwords even in cases one of the word is a suffix in some other word. Although, once IRSTLM is fixed, more examples can be analyzed which can benefit from the use of separable. @tanmai_khanna and @spectie were a huge help in guiding me through this part of the project :)
+===Tagger===
+The Hindi and Punjabi dictionaries were originally using taggers(.prob files) directly copied from the English module. @spectie helped me here to train a newer version and also making the taggers unigram.
+===Transfer Rules===
+Hindi and Punjabi are fairly even when it comes to syntax, so rules from the existing Urdu-Hindi module were mostly sufficient for this pair as well.
+===Quality===
+The translator performs good (tested on Wikipedia articles and stories taken from indic websites) as long as there aren't a lot of borrowed words or proper nouns with spelling variations. This remains to be challenge especially in the case of technical Wikipedia articles. On a generic Wiki article about the state of Pakistan, the translator achieved a WER of 10.84%.
 ==Current tasks==

@@ Line 229: / Line 229: @@
 ! 81%
 ! 8.4%, 6.78%
-! 26.44,22.31
+! 26.44%, 22.31%
 ! -
 ! WER, PER On a set of 180 sentences(3200 words)