User:Khannatanmai/GSoC2019Report - Revision history

Khannatanmai: /* Anaphora Resolution in Apertium */

2020-03-18T18:53:15Z

Anaphora Resolution in Apertium

Khannatanmai: /* Links to Work Done */

2019-08-26T12:47:28Z

Links to Work Done

Khannatanmai: /* Spanish - English */

2019-08-26T12:33:07Z

Spanish - English

Khannatanmai: /* Results */

2019-08-23T19:30:05Z

Results

Khannatanmai: /* Results */

2019-08-23T19:29:53Z

Results

Khannatanmai: /* Results */

2019-08-23T19:27:53Z

Results

Francis Tyers: /* Results */

2019-08-21T14:16:41Z

Results

Khannatanmai: /* Evaluation */

2019-08-21T10:08:23Z

Evaluation

Khannatanmai: /* Results */

2019-08-21T09:58:54Z

Results

Khannatanmai: /* Spanish - English */

2019-08-21T09:57:39Z

Spanish - English

@@ Line 1: / Line 1: @@
 == Anaphora Resolution in Apertium ==
-For this [https://summerofcode.withgoogle.com/projects/#5903412745469952 project] , I coded a new module for the Apertium Machine Translation Pipeline. The purpose of this module is to do Anaphora Resolution in the source text so that we can get a better translation output.
+For this [https://summerofcode.withgoogle.com/archive/2019/projects/5434868157120512/ project] , I coded a new module for the Apertium Machine Translation Pipeline. The purpose of this module is to do Anaphora Resolution in the source text so that we can get a better translation output.
 === Documentation ===

@@ Line 33: / Line 33: @@
 Pull Request: https://github.com/apertium/apertium-eng-spa/pull/13
+==== List of Commits made during GSoC to the above repositories ====
+https://apertium.projectjj.com/gsoc2019/khannatanmai/khannatanmai.html
 == Evaluation ==

@@ Line 41: / Line 41: @@
 Spanish has a possessive determiner ''su'', which can translate to ''his/her/its'' in English, so we need to resolve it as an anaphor.
-The Anaphora Resolution Module was run on a corpus of a 1000 sentences from Europarl, using this arx file: [Add link]
+The Anaphora Resolution Module was run on a corpus of a 1000 sentences from Europarl, using this [https://github.com/apertium/apertium-eng-spa/blob/anaphora-transfer/apertium-eng-spa.spa-eng.arx arx file]
 Out of these 1000 sentences, 258 sentences had at least one possessive determiner. The translations of these sentences with and without the Anaphora Resolution module in the pipeline were evaluated comparatively. The results are as follows:

@@ Line 85: / Line 85: @@
 '''Accuracy of Anaphora Resolution with the module on Spa-Eng: 54.17%'''
 '''Accuracy of Anaphora Resolution without the module on Spa-Eng: 29.17%'''
 '''Note''': Out of 258 sentences, 120 sentences had third person pronouns. The rest had first or second person pronouns which were anyway being translated correctly and are largely out of the scope of this module.

@@ Line 84: / Line 84: @@
 |}
-'''Accuracy of the module on Spa-Eng: 54.17%'''
+'''Accuracy of Anaphora Resolution with the module on Spa-Eng: 54.17%'''
+'''Accuracy of Anaphora Resolution without the module on Spa-Eng: 29.17%'''
 '''Note''': Out of 258 sentences, 120 sentences had third person pronouns. The rest had first or second person pronouns which were anyway being translated correctly and are largely out of the scope of this module.

@@ Line 67: / Line 67: @@
 | '''2'''
 |}
-<pre>
-P = 32 / 32 + 2 = 0.94
-R = 32 / 32 + 53 = 0.37
-F1 = (0.94 * 0.37) / (0.94 + 0.37) = 0.26
-</pre>
 Number of anaphors translated correctly without the Anaphora Resolution module and with:
@@ Line 89: / Line 83: @@
 | '''65'''
 |}
+'''Accuracy of the module on Spa-Eng: 54.17%'''
 '''Note''': Out of 258 sentences, 120 sentences had third person pronouns. The rest had first or second person pronouns which were anyway being translated correctly and are largely out of the scope of this module.

@@ Line 96: / Line 96: @@
 === Catalan - Italian ===
+A corpus was created from a freely available journal, and random paragraphs were analysed.
+In total, 108 cases of anaphora for the 3rd person possessive determiner in Catalan when translating it to Italian were analysed. What matters in this case is the number of the referent, but not his/her/its gender. Without anaphora, the referent is always chosen to be singular.
+==== Results ====
+* No Change, Correct: Anaphora Resolution didn't change the anaphor and it is correct.
+* No Change, Incorrect: Anaphora Resolution didn't change the anaphor, and it is incorrect, i.e. it should have changed.
+* Change, Correct: Anaphora Resolution changed the anaphor and it is now correct (was incorrect earlier).
+* Change, Incorrect: Anaphora Resolution changed the anaphor and it is now incorrect. (was correct earlier)
+{| class="wikitable"
+|-
+! colspan="2" | No Change
+! colspan="2" | Change
+|-
+| Correct
+| Incorrect
+| Correct
+| Incorrect
+|-
+| '''76'''
+| '''13'''
+| '''5'''
+| '''14'''
+|}
+Number of anaphors translated correctly without the Anaphora Resolution module and with:
+{| class="wikitable"
+|-
+! Total 3rd Person Anaphors
+! Without Anaphora Resolution
+! With Anaphora Resolution
+|-
+|
+| Correct
+| Correct
+|-
+| '''108'''
+| '''90'''
+| '''81'''
+|}
+==== Observations ====
+* In this corpus, just choosing singular gives correct translations in 90/108 examples so the anaphors aren't evenly spread out.
+* While the Anaphora Resolution module gives worse results here, the configurations can be tuned to give much better results for this corpus.
+For detailed observations, refer to the [https://drive.google.com/file/d/18MSisDqrq0DDAHhzTkuBcsj9INURJu50/view?usp=sharing Complete Evaluation] and go to '''Catalan-Italian'''.
 == Future Ideas ==