Difference between revisions of "User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress"
Jump to navigation
Jump to search
(30 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | ==Current tasks(Week |
+ | ==Current tasks(Week 8)== |
+ | * Number of subject-object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ. मेरे(marked mf) | मेरा(marked m) |
||
− | * Fix some discrepancies due to postpositions(posttransfer) |
||
+ | * Fix सब__prn for खुद |
||
− | * Add Adverbs. |
||
+ | * Fix determiners in Hindi (Add काफी to determiners) |
||
− | * Clean up parallel text and compute WER. |
||
+ | * Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ. |
||
+ | * What is the analysis for ਦੋਸਤੋਂ. |
||
+ | * ਮੈਂ is the translation for मैंने and मैं. Add transfer rule and check if tagger correctly identifies which one is being picked. |
||
+ | * Fix alternate spellings analyses.. |
||
+ | * कि v/s की punjabi ki can also be kya. |
||
+ | * Fix gender for adj - ਬਹੁਤੀ |
||
+ | * Complete manual analysis of bidix entries for adjectives and adverbs. |
||
+ | * Add more verbs. |
||
+ | * Run testvoc on Adjectives and Adverbs. |
||
+ | |||
+ | * aisa <noun> - ajiha BUT aisa <verb> innj. |
||
== Progress == |
== Progress == |
||
Line 19: | Line 30: | ||
! hin |
! hin |
||
! pan |
! pan |
||
− | ! |
+ | ! pan-hin |
! hin |
! hin |
||
! pan |
! pan |
||
− | ! |
+ | ! pan-hin |
! hin→pan |
! hin→pan |
||
! pan→hin |
! pan→hin |
||
Line 40: | Line 51: | ||
! - |
! - |
||
! - |
! - |
||
+ | ! Original coverage - 12.8% |
||
− | ! - |
||
|- |
|- |
||
Line 62: | Line 73: | ||
! - |
! - |
||
! +500 |
! +500 |
||
− | ! |
+ | ! +250 |
! - |
! - |
||
! 58% |
! 58% |
||
Line 70: | Line 81: | ||
! - |
! - |
||
! WER, PER On a set of 50 sentences(1038 words) |
! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 4 |
||
+ | ! June 8-14 |
||
+ | ! +50 |
||
+ | ! +1500 |
||
+ | ! +1000 |
||
+ | ! - |
||
+ | ! 63% |
||
+ | ! - |
||
+ | ! 41.36%, 35.82% |
||
+ | ! 41.47%, 37.03% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 5 |
||
+ | ! June 15-21 |
||
+ | ! - |
||
+ | ! +50 |
||
+ | ! +200 |
||
+ | ! - |
||
+ | ! 66% |
||
+ | ! 52.8% |
||
+ | ! 41.36%, 35.82% |
||
+ | ! 41.47%, 37.03% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 6 |
||
+ | ! June 22-28 |
||
+ | ! - |
||
+ | ! +350 |
||
+ | ! +700 |
||
+ | ! - |
||
+ | ! 70% |
||
+ | ! 59% |
||
+ | ! 39.76%, 34.22% |
||
+ | ! 40.03%, 36.23% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 7 |
||
+ | ! June 29-Jul 6 |
||
+ | ! - |
||
+ | ! +350 |
||
+ | ! +700 |
||
+ | ! - |
||
+ | ! 71.3% |
||
+ | ! 61% |
||
+ | ! 39.76%, 34.22% |
||
+ | ! 40.03%, 36.23% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 8 |
||
+ | ! July 7-13 |
||
+ | ! - |
||
+ | ! +1200 |
||
+ | ! +1600 |
||
+ | ! - |
||
+ | ! 73% |
||
+ | ! 63% |
||
+ | ! 39.76%, 34.22% |
||
+ | ! 40.03%, 36.23% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 50 sentences(1038 words) |
||
+ | |- |
||
+ | |||
+ | ! 9 |
||
+ | ! July 14-20 |
||
+ | ! - |
||
+ | ! +2000 |
||
+ | ! +2000 |
||
+ | ! - |
||
+ | ! 74% |
||
+ | ! 65% |
||
+ | ! 36.76%, 32.22% |
||
+ | ! 39.03%, 33.53% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 100 sentences(2140 words) |
||
+ | |- |
||
+ | |||
+ | ! 10 |
||
+ | ! July 21-27 |
||
+ | ! - |
||
+ | ! +2500 |
||
+ | ! +2200 |
||
+ | ! - |
||
+ | ! 75% |
||
+ | ! 67% |
||
+ | ! 33.76%, 29.62% |
||
+ | ! 35.03%, 31.23% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 100 sentences(2140 words) |
||
+ | |- |
||
+ | |||
+ | ! 11 |
||
+ | ! July 28-Aug 2 |
||
+ | ! - |
||
+ | ! +1200 |
||
+ | ! +1300 |
||
+ | ! - |
||
+ | ! 78% |
||
+ | ! 70% |
||
+ | ! 32.34%, 28.82% |
||
+ | ! 34.45%, 30.43% |
||
+ | ! - |
||
+ | ! WER, PER On a set of 100 sentences(2140 words) |
||
|} |
|} |
||
===DONE=== |
===DONE=== |
||
+ | * DONE - Bidix updates. |
||
+ | * DONE - Fixed errors with postposition transfer. |
||
+ | * DONE - Added proper nouns. |
||
+ | * DONE - Added noun paradigms. |
||
+ | * DONE - Added Adverbs. |
||
* DONE - Added about 1400 adjective stems |
* DONE - Added about 1400 adjective stems |
||
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47% |
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47% |
||
Line 82: | Line 210: | ||
===IN PROGRESS=== |
===IN PROGRESS=== |
||
− | * IN PROGRESS - Adding |
+ | * IN PROGRESS - Adding Verbs. |
* IN PROGRESS - Fix inconsistencies with parallel texts. |
* IN PROGRESS - Fix inconsistencies with parallel texts. |
||
* IN PROGRESS - Practicing basic understanding of Persian script. |
* IN PROGRESS - Practicing basic understanding of Persian script. |
||
− | * IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms) |
||
− | ==TODO(Next week - Week |
+ | ==TODO(Next week - Week 3)== |
+ | * Add verbs and nouns. |
||
− | * Fix transfer rules for postpositions(fixing some issues). |
||
− | * Add verbs. |
||
− | * Get bidixes up-to-date with stems added to monodixes. |
||
==Literature(Apertium Wiki) Covered== |
==Literature(Apertium Wiki) Covered== |
Revision as of 12:28, 10 August 2020
Contents
Current tasks(Week 8)
- Number of subject-object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ. मेरे(marked mf) | मेरा(marked m)
- Fix सब__prn for खुद
- Fix determiners in Hindi (Add काफी to determiners)
- Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ.
- What is the analysis for ਦੋਸਤੋਂ.
- ਮੈਂ is the translation for मैंने and मैं. Add transfer rule and check if tagger correctly identifies which one is being picked.
- Fix alternate spellings analyses..
- कि v/s की punjabi ki can also be kya.
- Fix gender for adj - ਬਹੁਤੀ
- Complete manual analysis of bidix entries for adjectives and adverbs.
- Add more verbs.
- Run testvoc on Adjectives and Adverbs.
- aisa <noun> - ajiha BUT aisa <verb> innj.
Progress
Progress table
Week | Stems | Coverage | WER,PER | Progress | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
№ | dates | hin | pan | pan-hin | hin | pan | pan-hin | hin→pan | pan→hin | Evaluation | Notes |
1 | May 1-24 | - | - | - | - | 47% | - | - | - | - | Original coverage - 12.8% |
2 | May 24-31 | - | +1400 | - | - | 52% | - | 36.83%, 33.84% | 40.03%, 36.93% | - | WER, PER On a set of 25 sentences(612 words) |
3 | June 1-7 | - | +500 | +250 | - | 58% | - | 49.52%, 42.71% | 48.65%, 44.03% | - | WER, PER On a set of 50 sentences(1038 words) |
4 | June 8-14 | +50 | +1500 | +1000 | - | 63% | - | 41.36%, 35.82% | 41.47%, 37.03% | - | WER, PER On a set of 50 sentences(1038 words) |
5 | June 15-21 | - | +50 | +200 | - | 66% | 52.8% | 41.36%, 35.82% | 41.47%, 37.03% | - | WER, PER On a set of 50 sentences(1038 words) |
6 | June 22-28 | - | +350 | +700 | - | 70% | 59% | 39.76%, 34.22% | 40.03%, 36.23% | - | WER, PER On a set of 50 sentences(1038 words) |
7 | June 29-Jul 6 | - | +350 | +700 | - | 71.3% | 61% | 39.76%, 34.22% | 40.03%, 36.23% | - | WER, PER On a set of 50 sentences(1038 words) |
8 | July 7-13 | - | +1200 | +1600 | - | 73% | 63% | 39.76%, 34.22% | 40.03%, 36.23% | - | WER, PER On a set of 50 sentences(1038 words) |
9 | July 14-20 | - | +2000 | +2000 | - | 74% | 65% | 36.76%, 32.22% | 39.03%, 33.53% | - | WER, PER On a set of 100 sentences(2140 words) |
10 | July 21-27 | - | +2500 | +2200 | - | 75% | 67% | 33.76%, 29.62% | 35.03%, 31.23% | - | WER, PER On a set of 100 sentences(2140 words) |
11 | July 28-Aug 2 | - | +1200 | +1300 | - | 78% | 70% | 32.34%, 28.82% | 34.45%, 30.43% | - | WER, PER On a set of 100 sentences(2140 words) |
DONE
- DONE - Bidix updates.
- DONE - Fixed errors with postposition transfer.
- DONE - Added proper nouns.
- DONE - Added noun paradigms.
- DONE - Added Adverbs.
- DONE - Added about 1400 adjective stems
- DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
- DONE - Collected parallel texts to calculate WER,PER etc..
- DONE - Added bidirectional dictionary(33k paradigms)
- DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
- DONE - Scraped all Wikipedia texts and made a combined frequency list.
- DONE - Frequency lists using WikiExtractor on latest dump.
IN PROGRESS
- IN PROGRESS - Adding Verbs.
- IN PROGRESS - Fix inconsistencies with parallel texts.
- IN PROGRESS - Practicing basic understanding of Persian script.
TODO(Next week - Week 3)
- Add verbs and nouns.
Literature(Apertium Wiki) Covered
- DONE - Calculating Coverage.
- DONE - A long introduction on Transfer Rules.
- DONE - Transfer Rules examples
- DONE - Wikipedia Dumps.
- DONE - Generating Frequency lists.
- DONE - Building Dictionaries#Monolingual.
- DONE - Evaluation.
- DONE - Extract.
- DONE - Monodix Basics
- DONE - Improved Corpus Based Paradigm Matching.
- DONE - Transliteration.
- DONE - Workflow reference.
- DONE - Tagger Tranining.
- DONE - Modes introduction.
- DONE - Apertium-viewer.