User:Khannatanmai/GSoC2019Report
Anaphora Resolution in Apertium
For this project , I coded a new module for the Apertium Machine Translation Pipeline. The purpose of this module is to do Anaphora Resolution in the source text so that we can get a better translation output.
Documentation
I have prepared a verbose documentation for this module which explains what Anaphora Resolution means, it's role in the Apertium pipeline, the algorithm used, how to use the module and how to add the module to a new language pair.
You can find this here: Anaphora Resolution Module
Final Module
Here is the final released version of the Anaphora Module. Follow the instructions in the README to install this module on your system and run it.
Repository of Module: https://github.com/apertium/apertium-anaphora
Work Done during GSoC 2019
The above module is the final version of the Anaphora Resolution module, complete with a build system.
You can refer to the issues to see implementation of features and comments as they were being implemented during GSoC: https://github.com/apertium/apertium-anaphora/issues
During GSoC, this repository was used: https://github.com/khannatanmai/apertium-anaphora (This repository is just to highlight the peripheral work done during GSoC)
Links to Work Done
Code for the module
Main Repository: https://github.com/apertium/apertium-anaphora
Changes made to Apertium Code
To accomodate for the new module in the pipeline.
Pull Request: https://github.com/apertium/apertium/pull/55
Changes made to Apertium Spanish-English Language Pair
To test the new module on the Spanish-English language pair.
Pull Request: https://github.com/apertium/apertium-eng-spa/pull/13
Evaluation
The Anaphora Resolution module was tested on multiple languages with some basic indicators. I'll be presenting the results of the evaluation, which was done manually.
Spanish - English
Spanish has a possessive determiner su, which can translate to his/her/its in English, so we need to resolve it as an anaphor.
The Anaphora Resolution Module was run on a corpus of a 1000 sentences from Europarl, using this arx file: [Add link]
Out of these 1000 sentences, 258 sentences had at least one possessive determiner. The translations of these sentences with and without the Anaphora Resolution module in the pipeline were evaluated comparatively. The results are as follows:
Results
- No Change, Correct: Anaphora Resolution didn't change the anaphor and it is correct.
- No Change, Incorrect: Anaphora Resolution didn't change the anaphor, and it is incorrect, i.e. it should have changed.
- Change, Correct: Anaphora Resolution changed the anaphor and it is now correct (was incorrect earlier).
- Change, Incorrect: Anaphora Resolution changed the anaphor and it is now incorrect. (was correct earlier)
| No Change | Change | ||
|---|---|---|---|
| Correct | Incorrect | Correct | Incorrect |
| 33 | 53 | 32 | 2 |
Number of anaphors translated correctly without the Anaphora Resolution module and with:
| Total 3rd Person Anaphors | Without Anaphora Resolution | With Anaphora Resolution |
|---|---|---|
| Correct | Correct | |
| 120 | 35 | 65 |
Accuracy of the module on Spa-Eng: 54.17%
Note: Out of 258 sentences, 120 sentences had third person pronouns. The rest had first or second person pronouns which were anyway being translated correctly and are largely out of the scope of this module.
Observations
- A lot of the errors are made because the tagger gives the singular tag to group nouns such as Parliament, Commission, Group. If this is fixed, the results should improve significantly.
- Since the module only outputs his/her/their right now, all the examples with its haven't been resolved. Adding this would improve the results as well.
- The indicators one uses are corpus dependent. This corpus has a dialogue and hence we added an impeding indicator to patterns such as: <NP> <comma>, as that NP is usually the addressee.
For detailed observations, refer to the Complete Evaluation
Catalan - Italian
A corpus was created from a freely available journal, and random paragraphs were analysed.
In total, 108 cases of anaphora for the 3rd person possessive determiner in Catalan when translating it to Italian were analysed. What matters in this case is the number of the referent, but not his/her/its gender. Without anaphora, the referent is always chosen to be singular.
Results
- No Change, Correct: Anaphora Resolution didn't change the anaphor and it is correct.
- No Change, Incorrect: Anaphora Resolution didn't change the anaphor, and it is incorrect, i.e. it should have changed.
- Change, Correct: Anaphora Resolution changed the anaphor and it is now correct (was incorrect earlier).
- Change, Incorrect: Anaphora Resolution changed the anaphor and it is now incorrect. (was correct earlier)
| No Change | Change | ||
|---|---|---|---|
| Correct | Incorrect | Correct | Incorrect |
| 76 | 13 | 5 | 14 |
Number of anaphors translated correctly without the Anaphora Resolution module and with:
| Total 3rd Person Anaphors | Without Anaphora Resolution | With Anaphora Resolution |
|---|---|---|
| Correct | Correct | |
| 108 | 90 | 81 |
Observations
- In this corpus, just choosing singular gives correct translations in 90/108 examples so the anaphors aren't evenly spread out.
- While the Anaphora Resolution module gives worse results here, the configurations can be tuned to give much better results for this corpus.
For detailed observations, refer to the Complete Evaluation and go to Catalan-Italian.
Future Ideas
There are several future ideas to make this module better that I'll be trying my hand on after GSoC.
These are all mentioned in the Issues in the Module Repository: https://github.com/apertium/apertium-anaphora/issues .
