User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress

Progress on [1]

Current tasks(Week 1)

• Fix some discrepancies due to postpositions(posttransfer)
• Add Adverbs.
• Clean up parallel text and compute WER.

DONE

• DONE - Added about 1400 adjective stems
• DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
• DONE - Collected parallel texts to calculate WER,PER etc..
• DONE - Added bidirectional dictionary(33k paradigms)
• DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
• DONE - Scraped all Wikipedia texts and made a combined frequency list.
• DONE - Frequency lists using WikiExtractor on latest dump.

IN PROGRESS

• IN PROGRESS - Adding Adverbs.
• IN PROGRESS - Fix inconsistencies with parallel texts.
• IN PROGRESS - Practicing basic understanding of Persian script.
• IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists(might not be so beneficial since Punajbi and Hindi share inflections for different word forms)

TODO(Next week - Community Bonding Week 3)

• Fix transfer rules for postpositions(fixing some issues).
• Start work on function words addition to dictionary.

Literature(Apertium Wiki) Covered

• DONE - Calculating Coverage.
• DONE - A long introduction on Transfer Rules.
• DONE - Transfer Rules examples
• DONE - Wikipedia Dumps.
• DONE - Generating Frequency lists.
• DONE - Building Dictionaries#Monolingual.
• DONE - Evaluation.
• DONE - Extract.
• DONE - Monodix Basics
• DONE - Improved Corpus Based Paradigm Matching.
• DONE - Transliteration.

Progress table

Week		Stems			Coverage			WER,PER		Progress
№	dates	hin	pan	hin-pan	hin	pan	hin-pan	hin→pan	pan→hin	Evaluation	Notes
1	May 1-24	-	-	-	-	47%	-	-	-	-	-
2	May 24-31	-	+1400	-	-	52%	-	36.83%, 33.84%	40.03%, 36.93%	-	WER, PER On a set of 25 sentences(612 words)