User:Pmodi/GSOC 2020 proposal: Hindi-Punjabi/progress

Progress on [1]

Current task(Community bonding Week 1)

Finalising Frequency lists & Learning Persian script

• DONE - Fixed bidirectional translation i.e. pan->hin(gave close to human translation for small test set, even though similar transfer rules were copied)
• DONE - Gathered a sufficiently long parallel corpus. Currently cleaning up the xml files and generating aligned, text files for the two languages.
• DONE - Scraped all Wikipedia texts and made a combined frequency list.
• DONE - Frequency lists using WikiExtractor on latest dump.
• IN PROGRESS - Practicing basic understanding of Persian script.
• IN PROGRESS - Trying to collect parallel texts to calculate WER.
• IN PROGRESS - Extracting stems and suffix/prefix patterns using the frequency lists.

TODO(Next week - Community Bonding Week 2)

• Fix transfer rules for postpositions.
• Start work on function words addition to dictionary.

Literature(Apertium Wiki) Covered

• DONE - Wikipedia Dumps.
• DONE - Generating Frequency lists.
• DONE - Building Dictionaries#Monolingual.
• DONE - Evaluation.