Difference between revisions of "User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress"

From Apertium
Jump to navigation Jump to search
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
Progress on [https://wiki.apertium.org/wiki/User:Pmodi/GSOC_2020_proposal:_Hindi-Punjabi]
  
−  
 
==Current tasks(Week 1)==
  
==Current tasks(Week 1)==
 
* Fix some discrepancies due to postpositions(posttransfer)
  
* Fix some discrepancies due to postpositions(posttransfer)
Line 6: Line 4:
 
* Clean up parallel text and compute WER.
  
* Clean up parallel text and compute WER.
   
−
===DONE===
 +
== Progress ==
  ⚫
=== Progress table ===
⚫
* DONE - Added about 1400 adjective stems
  
⚫
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
  
⚫
* DONE - Collected parallel texts to calculate WER,PER etc..
  
⚫
* DONE - Added bidirectional dictionary(33k paradigms)
  
⚫
* DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
  
⚫
* DONE - Scraped all Wikipedia texts and made a combined frequency list.
  
⚫
* DONE - Frequency lists using WikiExtractor on latest dump.
  
−  
⚫
===IN PROGRESS===
  
⚫
* IN PROGRESS - Adding Adverbs.
  
⚫
* IN PROGRESS - Fix inconsistencies with parallel texts.
  
⚫
* IN PROGRESS - Practicing basic understanding of Persian script.
  
⚫
* IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms)
  
−  
⚫
==TODO(Next week - Community Bonding Week 3)==
  
⚫
* Fix transfer rules for postpositions(fixing some issues).
  
−
* Start work on function words addition to dictionary.
  
−  
⚫
==Literature(Apertium Wiki) Covered==
  
⚫
* DONE - Calculating Coverage.
  
⚫
* DONE - A long introduction on Transfer Rules.
  
⚫
* DONE - Transfer Rules examples
  
⚫
* DONE - Wikipedia Dumps.
  
⚫
* DONE - Generating Frequency lists.
  
⚫
* DONE - Building Dictionaries#Monolingual.
  
⚫
* DONE - Evaluation.
  
−  
⚫
== Progress table ==
  
   
 
{|class=wikitable
  
{|class=wikitable
Line 75: Line 46:
 
! May 24-31
  
! May 24-31
 
! -
  
! -
−
! -
 +
! +1400
 
! -
  
! -
 
! -
  
! -
 
! 52%
  
! 52%
 
! -
  
! -
  +
! 36.83%, 33.84%
  +
! 40.03%, 36.93%
 
! -
  
! -
  +
! WER, PER On a set of 25 sentences(612 words)
−
! -
  
−
! -
  
−
! -
  
 
|}
  
|}
  +
  +
===DONE===
  ⚫
* DONE - Added about 1400 adjective stems
  ⚫
* DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
  ⚫
* DONE - Collected parallel texts to calculate WER,PER etc..
  ⚫
* DONE - Added bidirectional dictionary(33k paradigms)
  ⚫
* DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
  ⚫
* DONE - Scraped all Wikipedia texts and made a combined frequency list.
  ⚫
* DONE - Frequency lists using WikiExtractor on latest dump.
  +
  ⚫
===IN PROGRESS===
  ⚫
* IN PROGRESS - Adding Adverbs.
  ⚫
* IN PROGRESS - Fix inconsistencies with parallel texts.
  ⚫
* IN PROGRESS - Practicing basic understanding of Persian script.
  ⚫
* IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms)
  +
  ⚫
==TODO(Next week - Week 2)==
  ⚫
* Fix transfer rules for postpositions(fixing some issues).
  +
* Add verbs.
  +
* Get bidixes up-to-date with stems added to monodixes.
  +
  ⚫
==Literature(Apertium Wiki) Covered==
  ⚫
* DONE - Calculating Coverage.
  ⚫
* DONE - A long introduction on Transfer Rules.
  ⚫
* DONE - Transfer Rules examples
  ⚫
* DONE - Wikipedia Dumps.
  ⚫
* DONE - Generating Frequency lists.
  ⚫
* DONE - Building Dictionaries#Monolingual.
  ⚫
* DONE - Evaluation.
  +
* DONE - Extract.
  +
* DONE - Monodix Basics
  +
* DONE - Improved Corpus Based Paradigm Matching.
  +
* DONE - Transliteration.

Revision as of 20:02, 4 June 2020

Current tasks(Week 1)

  • Fix some discrepancies due to postpositions(posttransfer)
  • Add Adverbs.
  • Clean up parallel text and compute WER.

Progress

Progress table

Week Stems Coverage WER,PER Progress
№ dates hin pan hin-pan hin pan hin-pan hin→pan pan→hin Evaluation Notes
1 May 1-24 - - - - 47% - - - - -
2 May 24-31 - +1400 - - 52% - 36.83%, 33.84% 40.03%, 36.93% - WER, PER On a set of 25 sentences(612 words)

DONE

  • DONE - Added about 1400 adjective stems
  • DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
  • DONE - Collected parallel texts to calculate WER,PER etc..
  • DONE - Added bidirectional dictionary(33k paradigms)
  • DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
  • DONE - Scraped all Wikipedia texts and made a combined frequency list.
  • DONE - Frequency lists using WikiExtractor on latest dump.

IN PROGRESS

  • IN PROGRESS - Adding Adverbs.
  • IN PROGRESS - Fix inconsistencies with parallel texts.
  • IN PROGRESS - Practicing basic understanding of Persian script.
  • IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms)

TODO(Next week - Week 2)

  • Fix transfer rules for postpositions(fixing some issues).
  • Add verbs.
  • Get bidixes up-to-date with stems added to monodixes.

Literature(Apertium Wiki) Covered

  • DONE - Calculating Coverage.
  • DONE - A long introduction on Transfer Rules.
  • DONE - Transfer Rules examples
  • DONE - Wikipedia Dumps.
  • DONE - Generating Frequency lists.
  • DONE - Building Dictionaries#Monolingual.
  • DONE - Evaluation.
  • DONE - Extract.
  • DONE - Monodix Basics
  • DONE - Improved Corpus Based Paradigm Matching.
  • DONE - Transliteration.
Retrieved from "https://wiki.apertium.org/w/index.php?title=User:Pmodi/GSOC_2020_proposal:_Hindi-Punjabi/progress&oldid=72330"