User:Khannatanmai/GSoC2020Proposal Trimming - Revision history

Khannatanmai: /* Non-Summer-Of-Code Plans */

2020-03-31T16:55:10Z

Non-Summer-Of-Code Plans

Khannatanmai: /* Skills and Qualifications */

2020-03-31T16:41:38Z

Skills and Qualifications

Khannatanmai: /* Which of the published tasks am I interested in? What do I plan to do? */

2020-03-31T16:40:10Z

Which of the published tasks am I interested in? What do I plan to do?

Khannatanmai: /* About Me */

2020-03-31T16:38:38Z

About Me

Khannatanmai: /* About Me */

2020-03-31T16:37:03Z

About Me

Khannatanmai: /* Work Plan */

2020-03-31T16:30:38Z

Work Plan

Khannatanmai: /* A description of how and who it will benefit in society */

2020-03-30T18:01:21Z

A description of how and who it will benefit in society

Khannatanmai: /* Skills and Qualifications */

2020-03-30T18:00:36Z

Skills and Qualifications

Khannatanmai: /* Work Plan (TODO) */

2020-03-30T14:05:10Z

Work Plan (TODO)

Khannatanmai: /* Work Plan (TODO) */

2020-03-30T14:00:25Z

Work Plan (TODO)

← Older revision		Revision as of 16:55, 31 March 2020
Line 331:		Line 331:

	I have no plans apart from GSoC in the summer and can devote 30-40 hours a week for this project.		I have no plans apart from GSoC in the summer and can devote 30-40 hours a week for this project.

			[[Category:GSoC_2020_student_proposals]]

@@ Line 320: / Line 320: @@
 I've been interested in linguistics from the very beginning and due to the rigorous programming courses, I'm also adept at several programming languages like Python, C++, XML, Bash Scripting, etc. I'm skilled in writing Finite State Transducers, Algorithms. Data Structures, and Machine Learning Algorithms as well.
-I also have a lot of experience studying and manually analysing data which. is an essential skill for doing problem solving in Linguistics
+I also have a lot of experience studying and manually analysing data which. is an essential skill for doing problem solving in Linguistics.
 I've worked with Apertium as part of GSoC 2019 and built the Anaphora resolution module, and hence I'm familiar with the codebase, writing parsers for the stream format, and the community which will help me to dive right in the project and make a significant contribution right from the start. I have worked in several other projects, such as a tool that predicts commas and sentence boundaries in ASR output using pitch, building a Translation Memory, Detecting Homographic Puns, POS Taggers, Grammar and Spell Checkers, Named Entity Recognisers, Building Chatbots, etc. all of which required a working understanding of Natural Language Processing. Most of these projects were done offline in my research lab and aren't available on GitHub because of the privacy settings but can be provided if needed.

@@ Line 45: / Line 45: @@
 = Project Proposal =
 == Which of the published tasks am I interested in? What do I plan to do? ==
-The task I'm interested in is '''Eliminating Dictionary Trimming'''([[Ideas_for_Google_Summer_of_Code/Eliminate_trimming]]). Dictionary trimming is the process of removing those words and their analyses from monolingual language models (FSTs compiled from monodixes) which don't have an entry in the bidix, to avoid a lot of untranslated lemmas (with an @ if debugging) in the output, which lead to issues with comprehension and post-editing the output. The section #WhyWeTrim explains the rationale behind trimming further.
+The task I'm interested in is '''Eliminating Dictionary Trimming'''([[Ideas_for_Google_Summer_of_Code/Eliminate_trimming]]). Dictionary trimming is the process of removing those words and their analyses from monolingual language models (FSTs compiled from monodixes) which don't have an entry in the bidix, to avoid a lot of untranslated lemmas (with an @ if debugging) in the output, which lead to issues with comprehension and post-editing the output. [[Why we trim]] explains the rationale behind trimming further.
 However, by trimming the dictionary, you throw away valuable analyses of words in the source language, which, if preserved, can be used as context for lexical selection and analysis of the input. Also, several transfer rules don't match as the word is shown as unknown. There are several other project ideas that would become viable if we don't trim away the analyses of monodix words that aren't in the bidix, such as a morph-guessing module that can output a source language lemma with target language morphology. Eliminating trimming would also help as if we learn morphological data for a language from an external source, if we continue to trim, it is unusable until all those words are in the bidix. As a general principle, it is better to try and not discard useful information, to pass it in the pipeline and then work on it based on the task.
-As part of this project, I plan to eliminate monodix trimming and propose a solution such that we don't lose the benefits of trimming. In the next section I will be describing a proposed solution as well as how I plan to work around everything in [[Why_we_trim]].
+As part of this project, I plan to eliminate monodix trimming and propose a solution such that we don't lose the benefits of trimming.
 == Proposed Workaround ==

@@ Line 27: / Line 27: @@
 '''What I want to get out of GSoC'''
-I've worked with Apertium as part of GSoC 2019 and I am familiar with the codebase and the community so I can do a lot more for Apertium during this GSoC and I want to be able to benefit the developers and users of the MT platform.
+I've worked with Apertium as part of GSoC 2019 and I am familiar with the codebase and the community so I can do a lot more for Apertium during this GSoC and with this project I'll be able to benefit the developers and users of the MT platform and develop a deep knowledge about the pipeline.
 Computational Linguistics is my passion, and I would love to work with similarly passionate people at Apertium, to develop tools that people actually benefit from. This would be an invaluable experience that classes just can't match.

@@ Line 27: / Line 27: @@
 '''What I want to get out of GSoC'''
+I've worked with Apertium as part of GSoC 2019 and I am familiar with the codebase and the community so I can do a lot more for Apertium during this GSoC and I want to be able to benefit the developers and users of the MT platform.
-I’ve enjoyed using Apertium in various personal and academic projects and it’s amazing to me that I get an opportunity to work with them.
 Computational Linguistics is my passion, and I would love to work with similarly passionate people at Apertium, to develop tools that people actually benefit from. This would be an invaluable experience that classes just can't match.

@@ Line 252: / Line 252: @@
 * Analyse the code of the parsers of the modules
-'''Phase 1: Week 1-4''' (June 1 - July 3)
+'''Phase 1'''(June 1 - July 3)
+'''Week 1'''
 * Write tests for each module to parse the new stream format
+'''Week 2,3,4'''
 * Modify the code for the parsers one by one in branches
 * Document the computational burden due to a longer stream
@@ Line 261: / Line 264: @@
 '''Evaluation 1: June 29'''
-'''Phase 2: Week 5-8''' (July 3 - July 27)
+'''Phase 2''' (July 3 - July 27)
+'''Week 5'''
 * Write tests for the tokeniser to add the surface form in the stream as well as the lemma and analysis
 * Write tests for the bidix lookup/generator to read surface form and output it if word missing in bidix
+'''Week 6'''
 * Modify tokeniser code
 * Modify bidix look up
+'''Week 7'''
 * Modify generator
 * Write tests for checking if trimming not needed
+'''Week 8'''
 * Add surface form in the stream as secondary information
 * Document elimination of trimming
@@ Line 275: / Line 283: @@
 '''Evaluation 2: July 27'''
-'''Phase 3: Week 9-12'''  (July 31 - August 24)
+'''Phase 3'''  (July 31 - August 24)
+'''Week 9'''
 * Test several language pairs that use trimming and run experiments and tests to ensure no regression of benefits
 * Test the new stream format with legacy data files and old language pairs
+'''Week 10'''
 * Document and prove without doubt that this is fully backwards compatible
 * Try solving markup reordering problem by putting markups as secondary info
+'''Week 11,12'''
 * Make a naive morph guessing module as described above (stretch goal)
 * Finish writing a thorough documentation and tests

@@ Line 291: / Line 291: @@
 = A description of how and who it will benefit in society =
-It will definitely benefit most users of Apertium and hopefully will attract more people to the tool. By discarding the morphological analysis of the word too early we prevent modules like lexical selection and transfer from using it, which would really benefit from this information. Secondly, as I've described earlier, there are several projects and ideas that would be possible once trimming is eliminated, because now the developer will have the option to use this morphological information anywhere in the pipeline. The morph guessing project will make the output of translation from Apertium much for comprehensive and would help with both gisting translation and post-editing, and hence help all kinds of users of Apertium.
+It will definitely benefit most users of Apertium and hopefully will attract more people to the tool. By discarding the morphological analysis of the word too early we prevent modules like lexical selection and transfer from using it, which would really benefit from this information. Secondly, as I've described earlier, there are several projects and ideas that would be possible once trimming is eliminated, because now the developer will have the option to use this morphological information anywhere in the pipeline. The morph guessing project will make the output of translation from Apertium much more comprehensive and would help with both gisting translation and post-editing, and hence help all kinds of users of Apertium.
 By modifying the stream to include arbitrary information, we also open the doors for several future applications, which can now add and use any info in the pipeline. This would be useful for markup reordering, semantic information, capitalisation, theta roles, sentiment tags, etc.

@@ Line 238: / Line 238: @@
 '''Now for the trimming, In the fourth statement we do a compose-intersect between the automorf fst (compiled monodix) and the autobil prefixes that we prepared above (modified bidix).''' Then, using <code>hfst-fst2fst</code> it converts the fst to an optimized-lookup (weighted) implementation and the final trimmed file is named <code>$(PREFIX1).automorf.hfst</code>. The unmodified automorf is put in the same optimised-lookup implementation and named <code>$(PREFIX1).automorf-untrimmed.hfst</code>. Note that <code>PREFIX1=$(LANG1)-$(LANG2)</code>.
-= Work Plan (TODO) =
+= Work Plan =
-'''Application Review Period''' (April 1 - May 4)
+'''Application Review Period''' (March 31 - May 3)
 * Document modification to Apertium stream format
 * Proof of Concept for the new format
@@ Line 252: / Line 252: @@
 * Analyse the code of the parsers of the modules
-'''Phase 1: Week 1-4''' (June 1 - )
+'''Phase 1: Week 1-4''' (June 1 - July 3)
 * Write tests for each module to parse the new stream format
 * Modify the code for the parsers one by one in branches
@@ Line 259: / Line 259: @@
 == Deliverable #1: Stream Modified such that the new format passes through each part of the pipeline ==
-'''Evaluation 1: June '''
+'''Evaluation 1: June 29'''
-'''Phase 2: Week 5-8''' (June )
+'''Phase 2: Week 5-8''' (July 3 - July 27)
 * Write tests for the tokeniser to add the surface form in the stream as well as the lemma and analysis
 * Write tests for the bidix lookup/generator to read surface form and output it if word missing in bidix
@@ Line 273: / Line 273: @@
 == Deliverable #2: Trimming eliminated without regression of benefits  ==
-'''Evaluation 2: July '''
+'''Evaluation 2: July 27'''
-'''Phase 3: Week 9-12'''  (July )
+'''Phase 3: Week 9-12'''  (July 31 - August 24)
 * Test several language pairs that use trimming and run experiments and tests to ensure no regression of benefits
 * Test the new stream format with legacy data files and old language pairs
@@ Line 283: / Line 283: @@
 * Finish writing a thorough documentation and tests
-'''Final Evaluations: August 24-'''
+'''Final Evaluations: August 24 - August 31'''
 == Project Completed ==