User:Agneet42/proposal - Revision history

Agneet42 at 13:52, 3 April 2017

2017-04-03T13:52:50Z

Agneet42 at 13:34, 3 April 2017

2017-04-03T13:34:56Z

Agneet42 at 13:31, 3 April 2017

2017-04-03T13:31:42Z

Agneet42 at 13:26, 3 April 2017

2017-04-03T13:26:12Z

Agneet42 at 13:15, 3 April 2017

2017-04-03T13:15:30Z

Agneet42 at 13:13, 3 April 2017

2017-04-03T13:13:43Z

Agneet42 at 13:10, 3 April 2017

2017-04-03T13:10:31Z

Agneet42 at 13:08, 3 April 2017

2017-04-03T13:08:08Z

Agneet42 at 13:06, 3 April 2017

2017-04-03T13:06:45Z

Agneet42 at 22:59, 2 April 2017

2017-04-02T22:59:13Z

@@ Line 59: / Line 59: @@
 # Add a constraint grammar which deals effectively with tenses/prepositions/verbs. CG should also handle other-POS ambiguities.
 # Tagger training to solve inflection issues.
+==Note==
+Since majority of the nominal lexical items in a language is an open set of words so it’s very difficult to include all the roots of Bengali in the Dictionary file at one go. The roots can always be increased with time leading to the automatic increase in the level of accuracy value.
 ==Work Plan==

@@ Line 142: / Line 142: @@
 # Work on transfer rules.
+# Continue work on Constraint Grammar.
 # Work up on chunking.[t2x]
 # Start with post-chunking.[t3x]

@@ Line 169: / Line 169: @@
 # Entries added into both the monolingual and bilingual dictionaries.
 # Taking into consideration the Bengali nouns, all the nominal suffixes have been identified. It has been noticed that the Bengali nominal suffixes consist of the classifiers, case markers and emphatic markers.
+# Working on the bengali translation of this[http://www.unilang.org/ulrview.php?res=394,405].
 =Other commitments=

@@ Line 44: / Line 44: @@
 Hindi and Bengali both originated from Old Indo-Aryan family of languages and are similar in structure. They have lot of similarities even though there are differences in the form of uses and positions of the words in corresponding sentences. Hindi pronouns can be broadly categorized into seven types namely, Personal,Demonstrative, Indefinite, Relative-Correlative, Possessive, Interrogative and Reflexive. Among these Hindi pronouns some are used both as Personal, Demonstrative, and Relative-Correlative pronouns. In Bengali, there are different pronouns for each of these uses. As the list of Hindi such pronouns is small and their uses are limited, it is possible to differentiate each use and find their Bengali translations using a set of linguistic rules. Some Hindi pronouns are used to demonstrate both animate and inanimate nouns and as third person personal pronouns. For these three uses a single Hindi pronoun is used where in Bengali there are dedicated pronouns for each use. Given such a Hindi pronoun, we have to find its use in the corresponding sentence and translate it to corresponding Bengali pronoun.
+Normally when two stems join together the inflectional suffix of the first stem remains unspecified in the resulting compound word. For example, the compound word “mAmA-bArI” (মামা বাড়ি)is actually the word “mAmAr bAri” (মামার বাড়ি) where “r” is the inflectional suffix for stem “mAmA”. That “r” is deleted when compound word is formed. This is called inflection deletion in compound words. So, when a inflectional suffix is found at
+the end of a compound word, it is presumed to be the inflectional suffix of the compound word, and not the inflectional suffix of the last stem. If all compound words followed the above inflection deletion then we should conclude that there is just one inflectional suffix for every compound word.
 == Current scenario ==
@@ Line 54: / Line 56: @@
 # Hindi nominal suffixes को(ko),का(kA),से(se),पे(pe), etc. are also used with Hindi pronouns. Different uses of these Hindi pronominal suffixes have different Bengali translations. The most frequent corresponding Bengali      pronominal suffixes are কে(ke), র(ra),থেকে(theke),এ(e), etc. The suffixes which have different translations can be disambiguated with the help of rules.
+#
 # Add a constraint grammar which deals effectively with tenses/prepositions/verbs. CG should also handle other-POS ambiguities.
 # Tagger training to solve inflection issues.

@@ Line 37: / Line 37: @@
 Apertium makes sure that the above issues do not come in it's path, and that is what makes it a suitable developmental ground for this (or any other) language pair. Furthermore, a hindi-bengali translation will make it easier for translation of similar languages like bengali such as hindi-assamese and hindi-oriya.
-==How and who will benefit in society==
+==How and who will benefit in society?==
 The monolingual dictionaries can be used as a stemmer for any search engine for Hindi/Bengali. It could also used as a spell checker. The effect of these  in other applications like, anaphora resolution, question answering can also be explored. The hindi-bengali translation will also help in accurate translation of manuscripts that are widely present in both the languages and make available the culture of both the forums to each other.

@@ Line 161: / Line 161: @@
 =Coding Challenge=
-To be upgraded post-application period: https://github.com/agneet42/Apertium-Ben-Hin
+To be <u>upgraded</u> post-application period: https://github.com/agneet42/Apertium-Ben-Hin
 # Set up the working environment (installation and configuration).

@@ Line 103: / Line 103: @@
 ====Week 5====
+# Test for (@,#,*) and remove them.
 # Gather translational data with the use of parallel corpora.
 # WER expected to go down by 10-15%.
@@ Line 133: / Line 134: @@
 # Carry out supervised tagger training for the languages.
 # Work on Constraint Grammar and disambiguations.
+# Start with Chunking.
 ====Week 10====
 # Work on transfer rules.
-# Begin with chunking[T2x and T3x]
+# Work up on chunking.[t2x]
+# Start with post-chunking.[t3x]
 ====Week 11====

@@ Line 57: / Line 57: @@
 # Tagger training to solve inflection issues.
-=Work Plan=
+==Work Plan==
 ====Major Goals====
@@ Line 117: / Line 117: @@
 # Prepare a list of word sequences that frequently appear together for both the languages(using the apriori algorithm)
 # Add multiwords with translations to the dictionaries.
-# Work further on transfer rules.
+# Work further on transfer rules[verbs etc.]
 # Clean testvoc for all classes.
 # Expected results : WER ~ 30%, Coverage ~ 60%
@@ Line 129: / Line 129: @@
 ====Week 9====
+# Obtain hand-tagged training corpora.
-#
+# Work on tag definition files.
+# Carry out supervised tagger training for the languages.
+# Work on Constraint Grammar and disambiguations.
+====Week 10====
+# Work on transfer rules.
+# Begin with chunking[T2x and T3x]
+====Week 11====
+# Thorough Regression and Corpus Testing.
+# Check dictionaries manually to spot possible errors followed by bugfixes(if any)
+====Week 12====
+# Evaluation of results and documentation.
+# WER ~ <=20%, Coverage ~ 70%
+<u> Deliverable #3: Language pair ready for or close to trunk </u>
+=Skills and Qualifications=
+I am a second year undergrad pursuing Computer Science and Technology from Jadavpur University, Kolkata, India. I have a steady coding experience in C, C++, Java and Python. I am also comfortable with XML and bash scripting. I have worked with the the Python libraries of TensorFlow, numpy and OpenCV before. I have an interest and a passion for Natural Language Processing and it's variants and I have undertaken projects in the same domain. Word Recognition using LSTM's[https://github.com/agneet42/Word-Spotting], AudioQA[https://github.com/agneet42/Q4AMRE] and Deep Speech. Furthermore, in the near future I wish to undertake the following project/s : Entailment and VisualQA.
+When it comes to languages, I am a native Bengali who can read/write both English and Hindi, and have learned Spanish and French over the years.
+=Coding Challenge=
+To be upgraded post-application period: https://github.com/agneet42/Apertium-Ben-Hin
+=Other commitments=
+I have no other commitments this summer. I plan on giving around 42 hours per week to the project. My college commences from August, even after which I can give ~30 hours for the project. I also do not have any plans of an impending vacation.
+=References=
+# Translations of Ambiguous Hindi Pronouns to Possible Bengali Pronouns [http://www.aclweb.org/anthology/W12-5214]
+# Development of a morphological analyser for Bengali [http://www.mt-archive.info/FreeRBMT-2009-Faridee.pdf]
+# Bengali Noun Morphological Analyzer [http://ieeexplore.ieee.org/document/6637408/]
+# Morphological Analysis of Inflecting Compound Words in Bangla [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.524.4988&rep=rep1&type=pdf]
+# A long introduction to transfer rules [http://wiki.apertium.org/w/index.php?title=User:Ariessa/proposal&action=edit]
+# Chunking: A full example [http://wiki.apertium.org/wiki/Chunking:_A_full_example]
+# Monodix Basics [http://wiki.apertium.org/wiki/Monodix_basics]
+# Finding errors in dictionaries [http://wiki.apertium.org/wiki/Finding_errors_in_dictionaries]
+# Apertium: Official Documentation [https://wiki.apertium.org/w/images/d/d0/Apertium2-documentation.pdf]