Difference between revisions of "User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress"

From Apertium
Jump to navigation Jump to search
 
(43 intermediate revisions by the same user not shown)
Line 1: Line 1:
  +
==Current tasks(Week 8)==
Progress on [https://wiki.apertium.org/wiki/User:Pmodi/GSOC_2020_proposal:_Hindi-Punjabi]
 
  +
* Number of subject-object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ. मेरे(marked mf) | मेरा(marked m)
  +
* Fix सब__prn for खुद
  +
* Fix determiners in Hindi (Add काफी to determiners)
  +
* Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ.
  +
* What is the analysis for ਦੋਸਤੋਂ.
  +
* ਮੈਂ is the translation for मैंने and मैं. Add transfer rule and check if tagger correctly identifies which one is being picked.
  +
* Fix alternate spellings analyses..
  +
* कि v/s की punjabi ki can also be kya.
  +
* Fix gender for adj - ਬਹੁਤੀ
  +
* Complete manual analysis of bidix entries for adjectives and adverbs.
  +
* Add more verbs.
  +
* Run testvoc on Adjectives and Adverbs.
   
  +
== Progress ==
==Current task(Community bonding Week 4)==
 
  +
=== Progress table ===
Fix some discrepancies due to postpositions(posttransfer) and add Adjectives, Adverbs.
 
 
===DONE===
 
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
 
* DONE - Collected parallel texts to calculate WER,PER etc..
 
* DONE - Added bidirectional dictionary(33k paradigms)
 
* DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
 
* DONE - Scraped all Wikipedia texts and made a combined frequency list.
 
* DONE - Frequency lists using WikiExtractor on latest dump.
 
 
===IN PROGRESS===
 
* IN PROGRESS - Adding Adjectives(~1300 stems) and Adverbs.
 
* IN PROGRESS - Fix inconsistencies with parallel texts.
 
* IN PROGRESS - Practicing basic understanding of Persian script.
 
* IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms)
 
 
==TODO(Next week - Community Bonding Week 3)==
 
* Fix transfer rules for postpositions(fixing some issues).
 
* Start work on function words addition to dictionary.
 
 
==Literature(Apertium Wiki) Covered==
 
* DONE - Calculating Coverage.
 
* DONE - A long introduction on Transfer Rules.
 
* DONE - Transfer Rules examples
 
* DONE - Wikipedia Dumps.
 
* DONE - Generating Frequency lists.
 
* DONE - Building Dictionaries#Monolingual.
 
* DONE - Evaluation.
 
 
== Progress table ==
 
   
 
{|class=wikitable
 
{|class=wikitable
Line 45: Line 28:
 
! hin
 
! hin
 
! pan
 
! pan
! hin-pan
+
! pan-hin
 
! hin
 
! hin
 
! pan
 
! pan
! hin-pan
+
! pan-hin
 
! hin→pan
 
! hin→pan
 
! pan→hin
 
! pan→hin
Line 66: Line 49:
 
! -
 
! -
 
! -
 
! -
  +
! Original coverage - 12.8%
  +
|-
  +
  +
! 2
  +
! May 24-31
 
! -
 
! -
  +
! +1400
  +
! -
  +
! -
  +
! 52%
  +
! -
  +
! 36.83%, 33.84%
  +
! 40.03%, 36.93%
  +
! -
  +
! WER, PER On a set of 25 sentences(612 words)
  +
|-
  +
  +
  +
! 3
  +
! June 1-7
  +
! -
  +
! +500
  +
! +250
  +
! -
  +
! 58%
  +
! -
  +
! 49.52%, 42.71%
  +
! 48.65%, 44.03%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 4
  +
! June 8-14
  +
! +50
  +
! +1500
  +
! +1000
  +
! -
  +
! 63%
  +
! -
  +
! 41.36%, 35.82%
  +
! 41.47%, 37.03%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 5
  +
! June 15-21
  +
! -
  +
! +50
  +
! +200
  +
! -
  +
! 66%
  +
! 52.8%
  +
! 41.36%, 35.82%
  +
! 41.47%, 37.03%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 6
  +
! June 22-28
  +
! -
  +
! +350
  +
! +700
  +
! -
  +
! 70%
  +
! 59%
  +
! 39.76%, 34.22%
  +
! 40.03%, 36.23%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 7
  +
! June 29-Jul 6
  +
! -
  +
! +350
  +
! +700
  +
! -
  +
! 71.3%
  +
! 61%
  +
! 39.76%, 34.22%
  +
! 40.03%, 36.23%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 8
  +
! July 7-13
  +
! -
  +
! +1200
  +
! +1600
  +
! -
  +
! 73%
  +
! 63%
  +
! 39.76%, 34.22%
  +
! 40.03%, 36.23%
  +
! -
  +
! WER, PER On a set of 50 sentences(1038 words)
  +
|-
  +
  +
! 9
  +
! July 14-20
  +
! -
  +
! +2000
  +
! +2000
  +
! -
  +
! 74%
  +
! 65%
  +
! 36.76%, 32.22%
  +
! 39.03%, 33.53%
  +
! -
  +
! WER, PER On a set of 100 sentences(2140 words)
  +
|-
  +
  +
! 10
  +
! July 21-27
  +
! -
  +
! +2500
  +
! +2200
  +
! -
  +
! 75%
  +
! 67%
  +
! 33.76%, 29.62%
  +
! 35.03%, 31.23%
  +
! -
  +
! WER, PER On a set of 100 sentences(2140 words)
  +
|-
  +
  +
! 11
  +
! July 28-Aug 2
  +
! -
  +
! +1200
  +
! +1300
  +
! -
  +
! 78%
  +
! 70%
  +
! 32.34%, 28.82%
  +
! 34.45%, 30.43%
  +
! -
  +
! WER, PER On a set of 100 sentences(2140 words)
 
|}
 
|}
  +
  +
===DONE===
  +
* DONE - Bidix updates.
  +
* DONE - Fixed errors with postposition transfer.
  +
* DONE - Added proper nouns.
  +
* DONE - Added noun paradigms.
  +
* DONE - Added Adverbs.
  +
* DONE - Added about 1400 adjective stems
  +
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
  +
* DONE - Collected parallel texts to calculate WER,PER etc..
  +
* DONE - Added bidirectional dictionary(33k paradigms)
  +
* DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
  +
* DONE - Scraped all Wikipedia texts and made a combined frequency list.
  +
* DONE - Frequency lists using WikiExtractor on latest dump.
  +
  +
===IN PROGRESS===
  +
* IN PROGRESS - Adding Verbs.
  +
* IN PROGRESS - Fix inconsistencies with parallel texts.
  +
* IN PROGRESS - Practicing basic understanding of Persian script.
  +
  +
==TODO(Next week - Week 3)==
  +
* Add verbs and nouns.
  +
  +
==Literature(Apertium Wiki) Covered==
  +
* DONE - Calculating Coverage.
  +
* DONE - A long introduction on Transfer Rules.
  +
* DONE - Transfer Rules examples
  +
* DONE - Wikipedia Dumps.
  +
* DONE - Generating Frequency lists.
  +
* DONE - Building Dictionaries#Monolingual.
  +
* DONE - Evaluation.
  +
* DONE - Extract.
  +
* DONE - Monodix Basics
  +
* DONE - Improved Corpus Based Paradigm Matching.
  +
* DONE - Transliteration.
  +
* DONE - Workflow reference.
  +
* DONE - Tagger Tranining.
  +
* DONE - Modes introduction.
  +
* DONE - Apertium-viewer.

Latest revision as of 13:17, 3 August 2020

Current tasks(Week 8)[edit]

  • Number of subject-object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ. मेरे(marked mf) | मेरा(marked m)
  • Fix सब__prn for खुद
  • Fix determiners in Hindi (Add काफी to determiners)
  • Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ.
  • What is the analysis for ਦੋਸਤੋਂ.
  • ਮੈਂ is the translation for मैंने and मैं. Add transfer rule and check if tagger correctly identifies which one is being picked.
  • Fix alternate spellings analyses..
  • कि v/s की punjabi ki can also be kya.
  • Fix gender for adj - ਬਹੁਤੀ
  • Complete manual analysis of bidix entries for adjectives and adverbs.
  • Add more verbs.
  • Run testvoc on Adjectives and Adverbs.

Progress[edit]

Progress table[edit]

Week Stems Coverage WER,PER Progress
dates hin pan pan-hin hin pan pan-hin hin→pan pan→hin Evaluation Notes
1 May 1-24 - - - - 47% - - - - Original coverage - 12.8%
2 May 24-31 - +1400 - - 52% - 36.83%, 33.84% 40.03%, 36.93% - WER, PER On a set of 25 sentences(612 words)
3 June 1-7 - +500 +250 - 58% - 49.52%, 42.71% 48.65%, 44.03% - WER, PER On a set of 50 sentences(1038 words)
4 June 8-14 +50 +1500 +1000 - 63% - 41.36%, 35.82% 41.47%, 37.03% - WER, PER On a set of 50 sentences(1038 words)
5 June 15-21 - +50 +200 - 66% 52.8% 41.36%, 35.82% 41.47%, 37.03% - WER, PER On a set of 50 sentences(1038 words)
6 June 22-28 - +350 +700 - 70% 59% 39.76%, 34.22% 40.03%, 36.23% - WER, PER On a set of 50 sentences(1038 words)
7 June 29-Jul 6 - +350 +700 - 71.3% 61% 39.76%, 34.22% 40.03%, 36.23% - WER, PER On a set of 50 sentences(1038 words)
8 July 7-13 - +1200 +1600 - 73% 63% 39.76%, 34.22% 40.03%, 36.23% - WER, PER On a set of 50 sentences(1038 words)
9 July 14-20 - +2000 +2000 - 74% 65% 36.76%, 32.22% 39.03%, 33.53% - WER, PER On a set of 100 sentences(2140 words)
10 July 21-27 - +2500 +2200 - 75% 67% 33.76%, 29.62% 35.03%, 31.23% - WER, PER On a set of 100 sentences(2140 words)
11 July 28-Aug 2 - +1200 +1300 - 78% 70% 32.34%, 28.82% 34.45%, 30.43% - WER, PER On a set of 100 sentences(2140 words)

DONE[edit]

  • DONE - Bidix updates.
  • DONE - Fixed errors with postposition transfer.
  • DONE - Added proper nouns.
  • DONE - Added noun paradigms.
  • DONE - Added Adverbs.
  • DONE - Added about 1400 adjective stems
  • DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
  • DONE - Collected parallel texts to calculate WER,PER etc..
  • DONE - Added bidirectional dictionary(33k paradigms)
  • DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
  • DONE - Scraped all Wikipedia texts and made a combined frequency list.
  • DONE - Frequency lists using WikiExtractor on latest dump.

IN PROGRESS[edit]

  • IN PROGRESS - Adding Verbs.
  • IN PROGRESS - Fix inconsistencies with parallel texts.
  • IN PROGRESS - Practicing basic understanding of Persian script.

TODO(Next week - Week 3)[edit]

  • Add verbs and nouns.

Literature(Apertium Wiki) Covered[edit]

  • DONE - Calculating Coverage.
  • DONE - A long introduction on Transfer Rules.
  • DONE - Transfer Rules examples
  • DONE - Wikipedia Dumps.
  • DONE - Generating Frequency lists.
  • DONE - Building Dictionaries#Monolingual.
  • DONE - Evaluation.
  • DONE - Extract.
  • DONE - Monodix Basics
  • DONE - Improved Corpus Based Paradigm Matching.
  • DONE - Transliteration.
  • DONE - Workflow reference.
  • DONE - Tagger Tranining.
  • DONE - Modes introduction.
  • DONE - Apertium-viewer.