User:Srbhr/GSOC 2020 Proposal: Automatic PostEditing - Revision history

Srbhr: /* Skills and Knowledge */

2020-03-30T17:29:53Z

Skills and Knowledge

Srbhr: /* Edit-Distance */

2020-03-30T12:24:27Z

Edit-Distance

Srbhr: /* Language Pairs & Data */

2020-03-30T12:17:34Z

Language Pairs & Data

Srbhr: /* Week 9-12 (31st July to 24th August) */

2020-03-30T12:16:01Z

Week 9-12 (31st July to 24th August)

Srbhr: /* Edit-Distance */

2020-03-28T00:09:39Z

Edit-Distance

Srbhr: /* Edit-Distance */

2020-03-28T00:08:38Z

Edit-Distance

Srbhr: /* Edit-Distance */

2020-03-28T00:01:37Z

Edit-Distance

Srbhr: /* Week 9-12 (31st July to 24th August) */

2020-03-27T23:54:11Z

Week 9-12 (31st July to 24th August)

Srbhr: /* Week 5-8 (3rd July to 27th July) */

2020-03-27T23:51:03Z

Week 5-8 (3rd July to 27th July)

Srbhr: /* Week 5-8 (3rd July to 27th July) */

2020-03-27T23:50:23Z

Week 5-8 (3rd July to 27th July)

@@ Line 48: / Line 48: @@
    (Stalled due to sudden college closure coz of CORONA Virus Pandemic).
-'''Cources that I've taken during my college time (Only Relevant to project):'''
+'''Courses that I've taken during my college time (Only Relevant to project):'''
 . Mathematics (I-IV, it includes statistics, calculus(both),probability and linear algebra)

@@ Line 151: / Line 151: @@
 Edit-Distance Algorithms that can be used for this task:
+    Levenshtein Distance (The previous work was based on this, hence forth. This will be chosen to work with, below are some other for listing.)
-    Levenshtein Distance
     Damerau-Levenshtein Distance
     Jaro Distance

@@ Line 190: / Line 190: @@
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:04, 25 March 2020 (CET) Do you have (S,MT(S),PE(MT(S))) triplets in OPUS? Give one example.
+WikiMedia dumps are a part of OPUS, and it's available in the provided triplets. (See my coding challenge dataset for the same)
 ----

@@ Line 272: / Line 272: @@
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:08, 25 March 2020 (CET)Say how
 * Accuracy can tested againg by using Edit-Distance algorithms, as we will have a test set of PE and we'll compare with the improved translation and PE(MT(S)), and try to
-find our scale at what we have performed.
+find our scale at what we have performed.If the MT(S) and PE(MT(S))have less edit rate then, likely it'll be an improvement between the pairs.
 * Also, try human verification of the Translation process, by asking someone to volunteer from the Apertium Group to check some language pairs.
 * Improve the Algorithm based on this process.

@@ Line 168: / Line 168: @@
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:03, 25 March 2020 (CET)Please explain why you need to compare tokens. How would you apply cosine similarity if you've got no embeddings?
-The Need for finding the relevant operators over the synonyms, makes us want to compare the tags generated by Apertium.tagger object for the same, and see how many similar elements they have. (While we need to ignore the words that are truly related to the word in general, and take only those elements which are useful from the whole synset.for eg. take the word
+The Need for finding the relevant operators over the synonyms, makes us want to compare the tags generated by Apertium.tagger object for the same, and see how many similar elements they have. (While we need to ignore the words that are truly related to the word in general, and take only those elements which are useful from the whole synset.for eg.
-   sake, it's synset includes:- purpose, determination, aim.
+   Take the word sake, it's synset includes:- purpose, determination, aim.
    running morphological analysis on all 4 will yield sake-> noun, but purpose-> noun, verb. aim-> noun, verb. determination -> noun.
    for creating a new entry "ben" as a translation of sake in the Galician pair we need to add some data to it, hence by running a Token Comparision, we can extract noun
@@ Line 179: / Line 180: @@
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:10, 25 March 2020 (CET) I miss an indication of the range of applicability: for instance, it may only work when PE(MT(S)) and MT(S) are similar.
 The Post-edits and Machine Translations are similar, but only different in sentence length, and some words. The rest 60-70% of the sentence structure remains the same. See the wikimedia dumps for any language pair on this.

@@ Line 167: / Line 167: @@
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:03, 25 March 2020 (CET)Please explain why you need to compare tokens. How would you apply cosine similarity if you've got no embeddings?
+The Need for finding the relevant operators over the synonyms, makes us want to compare the tags generated by Apertium.tagger object for the same, and see how many similar elements they have. (While we need to ignore the words that are truly related to the word in general, and take only those elements which are useful from the whole synset)
 All these algorithms are available through two Python Libraries (and are optimised).
@@ Line 172: / Line 173: @@
 *Textdistance[[https://pypi.org/project/textdistance/]]
 --[[User:Mlforcada|Mlforcada]] ([[User talk:Mlforcada|talk]]) 16:10, 25 March 2020 (CET) I miss an indication of the range of applicability: for instance, it may only work when PE(MT(S)) and MT(S) are similar.
+The Post-edits and Machine Translations are similar, but only different in sentence length, and some words. The rest 60-70% of the sentence structure remains the same. See the wikimedia dumps for any language pair on this.
 ===Language Pairs & Data===

@@ Line 250: / Line 250: @@
 .Find out the missing translations, and the mistranslation by comparing from the Extracted Pairs, S, MT, PE.
 .Trying to streamline the above two explained process,
-.Use the code from Deltamachine that she has done for the Bel-Rus pair and try to make it work for more language pairs, Likely it'll be a starting point for working  over the algorithm. (Her work is similar to the rationale described by the Mentor Mikel).
+.Use the code from Deltamachine that she has done for the Bel-Rus pair and try to make it work for more language pairs, Likely it'll be a starting point for working
+  over the algorithm. (Her work is similar to the rationale described by the Mentor Mikel).

@@ Line 219: / Line 219: @@
 * This is the most crucial step for the Project, hence a whole work phase of 4 weeks is dedicated to this.
 * This whole process is Explained in Task Description (Above [6.2]).
+Considering the fact, that a word may or may not have the exact translation in the language, or maybe not in the present form.
+   S: The twisted cubic is most easily given parametrically as the image of the map.
+   MT: El va torçar cúbic és més fàcilment donat *parametrically com la imatge del mapa
+   PE: La cúbica torçada es defineix més fàcilment de forma paramètrica com la imatge de la aplicació.
+Now if you see, the word parametrically is unknown by MT, hence If we can find words that are beginning with * then we can extract them. (Some work on this is done by Deltamachine in one of her codes, need to extract that function from there. Likely improve that to fit in the new code.)
+Now, if we align these word triplets. (parametrically, *parametrically, parametrica). We can do a guess by first using the root form of parametric (using external library) and see if that is available in the MT engine. In this case, Parametric is available and can be used to convert, and add into the MT dictionary.
+   ^parametric/parametric<adj>$^
+And using this knowledge we can create a starting point for entering parametica into the Catalan pair, as
+   ^paramètric<adj>$
+But, as we dive in deep into the problem, we can find such pairs that ^paramètrica/paramètric<adj><f><sg>$ is already available in the MT Engine for Eng-Cat pair, we only need to add some rule about adverbs and their superlative, comparative forms . And given a pair had no translation we could have a starting point where we could have inserted the word.
+Given another example for this.
+Take the Galician pair in my proposal page:
+   S: Never engage in action for the sake of reward.
+   MT(S): Nunca comprometer en acción para o *sake de recompense.
+   PE(MT(S)): Nunca comprometer en acción para o ben de recompense.
+Here sake is not translated.
+But if we try to find the synonym for sake and see if that is available in Galician Pair,
+   S: Never engage in action for the purpose of reward.
+   MT(S): Nunca comprometer en acción para o propósito de recompensa.
+We get an entry point for ben, from purpose. As purpose is a synonym/similar for sake, and as it’s available in both the dictionaries. We can do try a token comparison of sake, purpose, and propósito. And create an entry point for ben in the Dictionary on the basis of matching tokens only. (Not adding anything extra from purpose, only the common tags/matching tags. For that we need to do a token-based comparison, to compare the tags generated by the apertium-tagger.)
+The Algorithm is an automation of these tasks itself.
+.To align the text, using L-D and Previous work from Deltamachine as improving upon that.
+.Find out the missing translations, and the mistranslation by comparing from the Extracted Pairs, S, MT, PE.
+.Trying to streamline the above two explained process,
+.Use the code from Deltamachine that she has done for the Bel-Rus pair and try to make it work for more language pairs, Likely it'll be a starting point for working  over the algorithm. (Her work is similar to the rationale described by the Mentor Mikel).
 '''Deliverable (27th July to 31st July)''': Data Generated from the Streamlined Algorithm in the form of monodix, bidix, Grammars, etc.