User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress

Current tasks(Week 8)

• Number of object matters for pronouns : ਮੇਰੀਆਂ | ਸਾਡੀ
• Fix सब__prn for खुद
• Fix determiners in Hindi (Add काफी to determiners)
• Add post-positions to definitions - ਹਸਪਤਾਲੋਂ = ਹਸਪਤਾਲ ਤੋਂ.
• What is the analysis for ਦੋਸਤੋਂ.
• ਮੈਂ is the translation for मैंने and मैं. Add transfer rule and check if tagger correctly identifies which one is being picked.
• Fix alternate spellings analyses..
• कि v/s की punjabi ki can also be kya.
• Fix gender for adj - ਬਹੁਤੀ
• Complete manual analysis of bidix entries for adjectives and adverbs.
• Add more verbs.
• Run testvoc on Adjectives and Adverbs.

Progress

Progress table

Week		Stems			Coverage			WER,PER		Progress
№	dates	hin	pan	hin-pan	hin	pan	hin-pan	hin→pan	pan→hin	Evaluation	Notes
1	May 1-24	-	-	-	-	47%	-	-	-	-	Original coverage - 12.8%
2	May 24-31	-	+1400	-	-	52%	-	36.83%, 33.84%	40.03%, 36.93%	-	WER, PER On a set of 25 sentences(612 words)
3	June 1-7	-	+500	+250	-	58%	-	49.52%, 42.71%	48.65%, 44.03%	-	WER, PER On a set of 50 sentences(1038 words)
4	June 8-14	+50	+1500	+1000	-	63%	-	41.36%, 35.82%	41.47%, 37.03%	-	WER, PER On a set of 50 sentences(1038 words)
5	June 15-21	-	+50	+200	-	66%	52.8%	41.36%, 35.82%	41.47%, 37.03%	-	WER, PER On a set of 50 sentences(1038 words)
6	June 22-28	-	+350	+700	-	70%	59%	39.76%, 34.22%	40.03%, 36.23%	-	WER, PER On a set of 50 sentences(1038 words)

DONE

• DONE - Bidix updates.
• DONE - Fixed errors with postposition transfer.
• DONE - Added proper nouns.
• DONE - Added noun paradigms.
• DONE - Added Adverbs.
• DONE - Added about 1400 adjective stems
• DONE - Function words(cnj, det, prn, post, gen_endings), Coverage > 47%
• DONE - Collected parallel texts to calculate WER,PER etc..
• DONE - Added bidirectional dictionary(33k paradigms)
• DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
• DONE - Scraped all Wikipedia texts and made a combined frequency list.
• DONE - Frequency lists using WikiExtractor on latest dump.

IN PROGRESS

• IN PROGRESS - Adding Verbs.
• IN PROGRESS - Fix inconsistencies with parallel texts.
• IN PROGRESS - Practicing basic understanding of Persian script.

TODO(Next week - Week 3)

• Add verbs and nouns.

Literature(Apertium Wiki) Covered

• DONE - Calculating Coverage.
• DONE - A long introduction on Transfer Rules.
• DONE - Transfer Rules examples
• DONE - Wikipedia Dumps.
• DONE - Generating Frequency lists.
• DONE - Building Dictionaries#Monolingual.
• DONE - Evaluation.
• DONE - Extract.
• DONE - Monodix Basics
• DONE - Improved Corpus Based Paradigm Matching.
• DONE - Transliteration.
• DONE - Workflow reference.
• DONE - Tagger Tranining.
• DONE - Modes introduction.
• DONE - Apertium-viewer.