User:Rroychoudhury/GSoC 2020 Proposal - Revision history

Rroychoudhury: /* Coding Challenge */

2020-03-31T14:49:38Z

Coding Challenge

Rroychoudhury: /* Workplan */

2020-03-31T14:44:37Z

Workplan

Rroychoudhury: /* Workplan */

2020-03-31T14:44:05Z

Workplan

Rroychoudhury: /* Workplan */

2020-03-31T14:42:55Z

Workplan

Rroychoudhury: /* Workplan */

2020-03-31T14:41:17Z

Workplan

Rroychoudhury: /* My Proposal */

2020-03-30T09:16:58Z

My Proposal

Rroychoudhury: /* Coding Challenge */

2020-03-30T07:08:28Z

Coding Challenge

Rroychoudhury: /* Workplan */

2020-03-29T03:55:25Z

Workplan

Rroychoudhury: /* Workplan */

2020-03-29T03:52:05Z

Workplan

Rroychoudhury: /* Workplan */

2020-03-29T03:51:29Z

Workplan

@@ Line 257: / Line 257: @@
 == '''Coding Challenge''' ==
+*'''Sentiment Analysis'''
-*Install Apertium (see Minimal installation from SVN)
+**The main part of the coding challenge I did was create a neural network based program for sentiment analysis based on character level. The characters were encoded into a vector and were used as embedding weight for training. It was fed into some sequence to sequence models like LSTM ,RNN and also trained on a convolutional neural network.
-*Go through the HOWTO
+*'''Coding Challenge for selected topic'''
-*Rewrote paradigms(bn monolingual) ,added words to monolingual dictionary, Built a bilingual dictionary, wrote transfer files(.t1x), Constraint grammar (.rlx files)
+**Install Apertium (see Minimal installation from SVN)
-*Wrote a code for sentiment prediction[https://github.com/RajarshiRoychoudhury/apertium-bn-hi Code can be found here]
+**Go through the HOWTO
+**Wrote paradigms in monolingual Bengali dictionary for words like “রয়েছে খেয়েছে”
+**Added words in monolingual begali and hindi dictionary
+**Wrote intrachunk rules (ben-hin.t1x hin-ben.t1x)
+**Wrote Constraint Grammar( ben.rlx , hin.rlx)
+[https://github.com/RajarshiRoychoudhury/apertium-bn-hi Code can be found here]
 == '''Why should Google and Apertium sponsor it?''' ==

@@ Line 232: / Line 232: @@
 | style="text-align:center" | 17 August - 23 August
 |
-*Adding constraint grammer to o apertium-ben based on sentiment analysis
+*Adding constraint grammer to apertium-ben based on sentiment analysis
-*Adding constraint grammar to o apertium-hin based on sentiment analysis
+*Adding constraint grammar to apertium-hin based on sentiment analysis
 *Expanding bidix

@@ Line 207: / Line 207: @@
 | style="text-align:center" | 1 August - 8 August
 |
-*Expanding lexical selection rules(hi)
+*Expanding lexical selection rules(hin)
 *Expanding transfer rules(hin-ben) work on .t2x file
 *Expanding bidix
@@ Line 220: / Line 220: @@
 | style="text-align:center" |  9 August - 16 August
 |
-*Expanding lexical selection rules(hi)
+*Expanding lexical selection rules(hin)
-*Expanding transfer rules(hi-bn) work on .t3x file
+*Expanding transfer rules(hin-ben) work on .t3x file
 *Expanding bidix
 *Testing on resources(hin-ben translation)

@@ Line 168: / Line 168: @@
 | style="text-align:center" | ~4,000
 | style="text-align:center" | <38%
-| style="text-align:center" | ~78% (bn)<br>~87% (hi)
+| style="text-align:center" | ~78% (ben)<br>~87% (hi)
 |-
 ! 6
@@ Line 202: / Line 202: @@
 | style="text-align:center" | ~7,500
 | style="text-align:center" | <33%
-| style="text-align:center" |~82%(ben-hin)<br>~90%(hin-ben)
+| style="text-align:center" |~82%(ben)<br>~90%(hin)
 |-
 ! 9
@@ Line 208: / Line 208: @@
 |
 *Expanding lexical selection rules(hi)
-*Expanding transfer rules(hi-bn) work on .t2x file
+*Expanding transfer rules(hin-ben) work on .t2x file
 *Expanding bidix
-*Testing on resources(hi-bn translation)
+*Testing on resources(hin-ben translation)
 *Documentation
@@ Line 223: / Line 223: @@
 *Expanding transfer rules(hi-bn) work on .t3x file
 *Expanding bidix
-*Testing on resources(hi-bn translation)
+*Testing on resources(hin-ben translation)
 *Testing by translating from Hindi to Bengali
 | style="text-align:center" | ~9,500
@@ Line 249: / Line 249: @@
 | style="text-align:center" |~10,500
 | style="text-align:center" | <28%
-| style="text-align:center" | ~87% (ben-hin)<br>~90% (hin-ben)
+| style="text-align:center" | ~87% (ben)<br>~90% (hin)
 |-
 |}

@@ Line 78: / Line 78: @@
 **In apertium, we have
 ***Monolingual Bengali dictionary from apertium-bn-en(Bengali-English)
-***Monolingual Hindi dictionary from apertium-hin,(Apertium-Hindi)which will act as monolingual dictionary for apertium-bn-hi(Apertium-Bangla-Hindi).Another significant resource is
+***Monolingual Hindi dictionary from apertium-hin,(Apertium-Hindi)which will act as monolingual dictionary for apertium-ben-hin(Apertium-Bangla-Hindi).Another significant resource is
 **For Bengali and Hindi we have Sentiwordnet which will give sentiment annotated data of ~9000 words on each, which will act as training data for sentiment classification and these **words also be incorporated in the dictionaries.
 **Besides, wikipedia dumps, online text resources are also available. A university professor of linguistics has agreed to review my *work.
@@ Line 113: / Line 113: @@
 |
 *Determine sentiment of existing lexicons
-*Making paradigms to incorporate words like “রয়েছিল”.”খেয়েছিল”
+*Making paradigms to incorporate words like পরেছিল পড়েছিল”
 *Incorporating sentiment tags in the existing and new lexicons
 *Expanding the bidix
@@ Line 124: / Line 124: @@
 | style="text-align:center" | 8 June- 15 June
 |
-*Start making lexical selection rules(bn)
+*Start making lexical selection rules(ben)
+*Start making transfer rules(ben-hin)[like treating locative,genitive,nominal and accusatory nouns, proper nouns, proper transfer for noun- ex: “বাগানে আছে-बगीचे में हैं”, also making gender-verb agreement]]
-*Start making transfer rules(bn-hi)
 *Expanding bidix
 | style="text-align:center" | ~1,400
@@ Line 134: / Line 135: @@
 | style="text-align:center" | 16 June - 23 June
 |
-*Expanding lexical selection rules(bn)
+*Expanding lexical selection rules(ben)
-*Expanding transfer rules(bn-hi),work on .t1x file
+*Expanding transfer rules(ben-hin),work on .t1x file[identifying and modifying transfer rules to deal with cases where sentiment tags are required]
 *Expanding bidix
 *Documentation
@@ Line 146: / Line 147: @@
 | style="text-align:center" | 24 June - 30 June
 |
-*Expanding lexical selection rules(bn)
+*Expanding lexical selection rules(ben)
-*Expanding transfer rules(bn-hi) work on .t2x file
+*Expanding interchunk rules(ben-hin) work on .t2x file[ deal with 3 word transfer]
 *Expanding bidix
 *Testing on resources(bn-hi translation)
@@ Line 157: / Line 158: @@
 | style="text-align:center" | 1 July - 8 July
 |
-*Expanding lexical selection rules(bn)
+*Expanding lexical selection rules(ben)
-*Expanding transfer rules(bn-hi) work on .t3x file
+*Expanding postchunk rules(ben-hin) work on .t3x file
 *Expanding bidix
-*Testing on resources(bn-hi translation)
+*Testing on resources(ben-hin translation)
 *Testing by Translating Bengali to Hindi
 *Documentation
@@ Line 183: / Line 184: @@
 | style="text-align:center" | 16 July - 23 July
 |
-*Start making lexical selection rules(hi)
+*Start making lexical selection rules(hin)
+*Start making transfer rules(hin-ben)[especially making rules  to deal with patterns like nouns+post->noun(loc/gen/obj/nom) “बगीचे में हैं-বাগানে আছে-” ]
-*Start making transfer rules(hi-bn)
 *Expanding bidix
 | style="text-align:center" | ~6,500
@@ Line 193: / Line 195: @@
 | style="text-align:center" | 24 July - 31 July
 |
-*Expanding lexical selection rules(hi)
+*Expanding lexical selection rules(hin)
-*Expanding transfer rules(hi-bn),work on .t1x file
+*Expanding transfer rules(hin-ben),work on .t1x file[designing transfer rules with sentiment analysis to deal with ambiguous/non-traditional cases]n
 *Expanding bidix
 *Documentation
@@ Line 200: / Line 202: @@
 | style="text-align:center" | ~7,500
 | style="text-align:center" | <33%
-| style="text-align:center" |~82%(bn-hi)<br>~90%(hi-bn)
+| style="text-align:center" |~82%(ben-hin)<br>~90%(hin-ben)
 |-
 ! 9
@@ Line 230: / Line 232: @@
 | style="text-align:center" | 17 August - 23 August
 |
-*Adding constraint grammer to apertium-bn
+*Adding constraint grammer to o apertium-ben based on sentiment analysis
-*Adding constraint grammar to apertium-hi
+*Adding constraint grammar to o apertium-hin based on sentiment analysis
 *Expanding bidix
 | style="text-align:center" | ~10,500
@@ Line 241: / Line 244: @@
 |
 *Finish pending tasks
+*Testing
+*Documentation
 *'''Final Evaluation(31st August)'''
 | style="text-align:center" |~10,500
 | style="text-align:center" | <28%
-| style="text-align:center" | ~87% (bn-hi)<br>~90% (hi-bn)
+| style="text-align:center" | ~87% (ben-hin)<br>~90% (hin-ben)
 |-
 |}

@@ Line 23: / Line 23: @@
 == '''My Proposal''' ==
-<p style="font-size:16px">'''Title''': IMPROVING MACHINE TRANSLATION WITH SENTIMENT PRESERVATION IN HINDI-BANGLA PAIR</p>
+<p style="font-size:16px">'''Title''': IMPROVING MACHINE TRANSLATION WITH SENTIMENT TAGS IN HINDI-BANGLA PAIR</p>
 <p style="font-size:16px">'''Some problems with Translation pair for Bangla-Hindi''':</p>
 Existing models for Bangla-Hindi are far from accurate. For example, when the following sentence written in Bengali is given in Google Translate, it gives a completely incorrect translation.<br>

@@ Line 79: / Line 79: @@
 ***Monolingual Bengali dictionary from apertium-bn-en(Bengali-English)
 ***Monolingual Hindi dictionary from apertium-hin,(Apertium-Hindi)
-which will act as monolingual dictionary for apertium-bn-hi(Apertium-Bangla-Hindi).Another significant resource is
+*which will act as monolingual dictionary for apertium-bn-hi(Apertium-Bangla-Hindi).Another significant resource is
-for Bengali and Hindi we have Sentiwordnet which will give sentiment annotated data of ~9000 words on each, which will act as training data for sentiment classification and these words also be incorporated in the dictionaries. Besides, wikipedia dumps, online text resources are also available. A university professor of linguistics has agreed to review my work.
+*for Bengali and Hindi we have Sentiwordnet which will give sentiment annotated data of ~9000 words on each, which will act as training data for sentiment classification and these *words also be incorporated in the dictionaries. Besides, wikipedia dumps, online text resources are also available. A university professor of linguistics has agreed to review my *work.
 *'''Calculation of word error rate(WER)''':