Automatic postediting at GSoC 2018 - Revision history

Deltamachine at 15:27, 15 August 2018

2018-08-15T15:27:32Z

Deltamachine at 13:13, 14 August 2018

2018-08-14T13:13:34Z

Deltamachine: /* Inserting operations into a language pair: separate module approach */

2018-08-14T11:27:05Z

Inserting operations into a language pair: separate module approach

Deltamachine at 13:24, 12 August 2018

2018-08-12T13:24:29Z

Deltamachine at 12:29, 12 August 2018

2018-08-12T12:29:43Z

Deltamachine: /* Inserting operations into a language pair */

2018-08-10T20:45:40Z

Inserting operations into a language pair

Deltamachine: /* Cleaning */

2018-08-09T20:28:12Z

Cleaning

Deltamachine: /* New algorithm for operations extraction */

2018-08-09T20:27:01Z

New algorithm for operations extraction

Deltamachine: /* Classifying operations */

2018-08-09T18:09:34Z

Classifying operations

Deltamachine at 09:27, 9 August 2018

2018-08-09T09:27:12Z

@@ Line 33: / Line 33: @@
 === Toolbox ===
-The toolbox and step-to-step guide about how to use it: https://github.com/deltamachine/naive-automatic-postediting/tree/master/new_alg
+The toolbox and step-to-step guide about how to use it: https://github.com/deltamachine/naive-automatic-postediting/tree/master/toolbox
 ==== New algorithm for operations extraction ====
-Because of meaningless results of using the old algorithm, the new algorithm was created. It is based on the custom alignment. It seems that the new code will work okay on close-related languages, but I'm not sure about others. The code can be found here: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/new_learn_postedits_algorithm.py and the rationale can be found here: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/rationale.md.
+Because of meaningless results of using the old algorithm, the new algorithm was created. It is based on the custom alignment. It seems that the new code will work okay on close-related languages, but I'm not sure about others. The code can be found here: https://github.com/deltamachine/naive-automatic-postediting/blob/master/toolbox/new_learn_postedits_algorithm.py and the rationale can be found here: https://github.com/deltamachine/naive-automatic-postediting/blob/master/toolbox/rationale.md.
 ==== Classifying operations ====
-A script for classifying extracted postedits (https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/extract_types.py) indentifies three types of operations: potential monodix/bidix entries (when a pair doesn't have a translation for a given word), grammar mistakes (when Apertium chooses incorrect form of translated word) and other mistakes (it can be, for example, a potential lexical selection rule).
+A script for classifying extracted postedits (https://github.com/deltamachine/naive-automatic-postediting/blob/master/toolbox/extract_types.py) indentifies three types of operations: potential monodix/bidix entries (when a pair doesn't have a translation for a given word), grammar mistakes (when Apertium chooses incorrect form of translated word) and other mistakes (it can be, for example, a potential lexical selection rule).
 How it works:
@@ Line 58: / Line 58: @@
 ==== Cleaning ====
-The extracting postedits algorithm is not perfect and extracts a lot of garbage along with potentially good triplets. For cleaning files with postedits a following script was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/clean_postedits.py. On the first step it tags every part of every triplet using apertium-tagger and then drops triplets which contain punctuation. It helps filter out wrong triplets like, for example, (',', '*видець', ',').
+The extracting postedits algorithm is not perfect and extracts a lot of garbage along with potentially good triplets. For cleaning files with postedits a following script was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/toolbox/clean_postedits.py. On the first step it tags every part of every triplet using apertium-tagger and then drops triplets which contain punctuation. It helps filter out wrong triplets like, for example, (',', '*видець', ',').
 Then it calculates the same metric as in classifying step between s and mt, mt and pe, s and pe. If every result >= 30 and triplet is not from "other mistakes" list, the algorithm saves this triplet, if not - drops it. It helps filter out wrong alignment cases.
@@ Line 79: / Line 79: @@
 ==== Inserting operations into a language pair: separate module approach (under development) ====
-A script for applying learned postedits to new sentences was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/new_apply_postedits.py
+A script for applying learned postedits to new sentences was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/toolbox/new_apply_postedits.py
 It applies postedits to a given test MT file and creates an output file which contains source (S), Apertium translated (MT), edited by algorithm (ED) and target sentences (T) in a following format:

@@ Line 31: / Line 31: @@
 There were three attempts to extract postediting operations for each language pair: with threshold = 0.8 and -m, -M = (1, 3).
 In fact, results are not very meaningful: the reason might lie in problems in ''learn_postedits.py'' and in the method itself (but it should be checked carefully).
+=== Toolbox ===
+The toolbox and step-to-step guide about how to use it: https://github.com/deltamachine/naive-automatic-postediting/tree/master/new_alg
 ==== New algorithm for operations extraction ====
@@ Line 59: / Line 62: @@
 Then it calculates the same metric as in classifying step between s and mt, mt and pe, s and pe. If every result >= 30 and triplet is not from "other mistakes" list, the algorithm saves this triplet, if not - drops it. It helps filter out wrong alignment cases.
-==== Inserting operations into a language pair: dictionary approach ====
+==== Inserting operations into a language pair: dictionary approach (under development) ====
 For inserting operations into a language pair, a few helper scripts were written.
@@ Line 75: / Line 78: @@
 . The last step is to run ''add_new_entries.py'' on the edited table. This script will create new antries, add them to the dictionaries and compile them.
-==== Inserting operations into a language pair: separate module approach ====
+==== Inserting operations into a language pair: separate module approach (under development) ====
 A script for applying learned postedits to new sentences was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/new_apply_postedits.py

@@ Line 107: / Line 107: @@
 |40.50% / 36.76%
 |44.09 / 37.36%
+|-
+|}
+Little experiment with Spanish - Catalan (with applying only the postedits from "potential bidix entries" list learned on train data).
+{| class = "wikitable" style = "background-color: white; text-align: center; width: 70%;"
+|-
+|
+|'''spa-cat'''
+|-
+|'''(MT, T) WER / position-independent WER'''
+|22.49% / 15.03%
+|-
+|'''(ED, T) WER / position independent WER'''
+|22.44% / 14.98%
 |-
 |}

← Older revision		Revision as of 13:24, 12 August 2018
Line 74:		Line 74:

	5. The last step is to run ''add_new_entries.py'' on the edited table. This script will create new antries, add them to the dictionaries and compile them.		5. The last step is to run ''add_new_entries.py'' on the edited table. This script will create new antries, add them to the dictionaries and compile them.


	==== Inserting operations into a language pair: separate module approach ====		==== Inserting operations into a language pair: separate module approach ====

@@ Line 59: / Line 59: @@
 Then it calculates the same metric as in classifying step between s and mt, mt and pe, s and pe. If every result >= 30 and triplet is not from "other mistakes" list, the algorithm saves this triplet, if not - drops it. It helps filter out wrong alignment cases.
-==== Inserting operations into a language pair ====
+==== Inserting operations into a language pair: dictionary approach ====
 For inserting operations into a language pair, a few helper scripts were written.
@@ Line 75: / Line 75: @@
 . The last step is to run ''add_new_entries.py'' on the edited table. This script will create new antries, add them to the dictionaries and compile them.
-==== Evaluation ====
+==== Inserting operations into a language pair: separate module approach ====
+A script for applying learned postedits to new sentences was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/new_apply_postedits.py
+It applies postedits to a given test MT file and creates an output file which contains source (S), Apertium translated (MT), edited by algorithm (ED) and target sentences (T) in a following format:
+S	я ненавижу спешить по утрам.
+MT	я *ненавижу *спешить по ранкам.
+ED	я ненавиджу поспішати по ранкам.
+T	я ненавиджу поспішати вранку.
+For testing this approach a fast-and-dirty WER checking script was written. It takes file which was created by ''new_apply_postedits.py'', collects all MT, ED and T sentences (in case of few ED for the same sentence, the first one is chosen) and runs apertium-eval-translator on (MT, T) and (ED, T).
+Here are the results for test data (with applying only the postedits from "potential bidix entries" list learned on train data).
+{| class = "wikitable" style = "background-color: white; text-align: center; width: 70%;"
+|-
+|
+|'''bel-rus'''
+|'''rus-ukr'''
+|-
+|'''(MT, T) WER / position-independent WER'''
+|42.48% / 38.74%
+|47.25% / 40.78%
+|-
+|'''(ED, T) WER / position independent WER'''
+|40.50% / 36.76%
+|44.09 / 37.36%
+|-
+|}

@@ Line 60: / Line 60: @@
 ==== Inserting operations into a language pair ====
+For inserting operations into a language pair, a few helper scripts were written.
+===== Monodix/bidix entries =====
+New monodix/bidix entries can be created from postedits in the following way:
+. Firstly, ''create_entries_table.py'' takes a file with bidix postedits, splits it in source and target, then analyzes both sides using UDPipe (in case of Belarusian and Ukranian) or Mystem (in case of Russian), finds a lemma of every word, replaces UD/Mystem tages with Apertium ones and then create a file, which contains table with rows "source lemma - source Apertium tag - target lemma - target Apertium tag".
+. After that, the table should be manually checked: UDPipe/Mystem not always determine a correct/lemma for a word.
+. Then ''check_entries.py'' should be run on the created table. This script looks for the given lemmas/pair of lemmas in source/target/bidix dictionaries and creates a new table with information for every word.
+. Then user should again manually edit the table and add a stem and a paradigm for every word, which was not found in dictionaries.
+. The last step is to run ''add_new_entries.py'' on the edited table. This script will create new antries, add them to the dictionaries and compile them.
 ==== Evaluation ====

@@ Line 41: / Line 41: @@
 ) It takes file with postedit triplets (s, mt, pe).
 ) If here is '*' in mt, algorithm adds triplet to "potential bidix entries" list.
 ) If not, the script calculates the following metric:
+''x = ((l - d) / l) * 100''
-* letters = number of letters in pe
-* distance = Levenshtein distance betweeen mt and pe
+where l = number of letters in pe and d = Levenshtein distance betweeen mt and pe.
+If 50 <= x < 100, the algorithm adds triplet to "grammar mistakes" list.
-((letters - distance) / letters) * 100
-If 50 <= this number < 100, the algorithm adds triplet to "grammar mistakes" list.
+) Otherwise the algorithm checks, if mt != pe, and if not, adds triplet to "other mistakes" list.
-) Else the algorithm checks, if mt != pe, and if no, adds triplet to "other mistakes" list.
 ==== Cleaning ====
 The extracting postedits algorithm is not perfect and extracts a lot of garbage along with potentially good triplets. For cleaning files with postedits a following script was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/clean_postedits.py. On the first step it tags every part of every triplet using apertium-tagger and then drops triplets which contain punctuation. It helps filter out wrong triplets like, for example, (',', '*видець', ',').

@@ Line 5: / Line 5: @@
 https://github.com/deltamachine/naive-automatic-postediting
-== Workplan ==
-{|class=wikitable
-|-
-! Week    !! Dates       !! To do
-|-
-| 1     || 14th May — 20th May  ||  <s>Find and download needed Russian - Ukranian and Russian - Belarusian corpora, write scripts for preprocessing the data.</s>
-|-
-| 2 || 21th May - 27th May || <s> Learn to use bicleaner (https://github.com/sortiz/bicleaner), train ru-uk classifier, preprocess OpenSubtitles corpora, filter out loose translations. </s>
-|-
-| 3     || 28th May — 3rd June || <s> Continue to prepare Russian - Ukranian parallel corpus from OpenSubtitles, refactore the old apply_postedits.py code, make the old code work faster. </s>
-|-
-| 4      || 4th June — 10th June  || Work on the old code, start to extract triplets.
-|-
-! '''First evaluation, 11th June - 15th June''' !!  colspan="2" align=left |
-|-
-| 5    || 11th June — 17th June ||
-|-
-| 6     || 18th Jule — 24th July ||
-|-
-| 7     || 25th July — 1st July  ||
-|-
-| 8     || 2nd July — 8th July  ||
-|-
-!'''Second evaluation, 9th July - 13th July''' ||  colspan="2" align=left |
-|-
-| 9     || 9th July — 15th July ||
-|-
-| 10      || 16th July — 22th July   ||
-|-
-| 11      || 23rd July — 29th July  ||
-|-
-| 12      || 30th August — 5th August  ||
-|-
-!'''Final evaluation, 6th August - 14th August''' ||  colspan="2" align=left |
-|-
-|}
 == Progress notes ==
@@ Line 78: / Line 25: @@
 Total amount of sentences: 8463.
 ==== Code refactoring ====
 Two old scripts, ''learn_postedits.py'' and ''apply_postedits.py'' were refactored. Also now both scripts work approximately 10 times faster: now scripts collect all subsegments in one large file and translate/analyze the whole file. Instead of calling Apertium few times for every subsegment, now it is called only two times (for translating and for analyzing) for all subsegments of a sentence.
@@ Line 85: / Line 31: @@
 There were three attempts to extract postediting operations for each language pair: with threshold = 0.8 and -m, -M = (1, 3).
 In fact, results are not very meaningful: the reason might lie in problems in ''learn_postedits.py'' and in the method itself (but it should be checked carefully).
+==== New algorithm for operations extraction ====
+Because of meaningless results of using the old algorithm, the new algorithm was created. It is based on the custom alignment algorithm. It seems that the new code will work okay on close-related languages, but I'm not sure about others. The code and rationale can be found here: https://github.com/deltamachine/naive-automatic-postediting/tree/master/new_alg
+==== Classifying operations ====
+A script for classifying extracted postedits (https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/extract_types.py) indentifies three types of operations: potential monodix/bidix entries (when a pair doesn't have a translation for a given word), grammar mistakes (when Apertium chooses incorrect form of translated word) and other mistakes (it can be, for example, a potential lexical selection rule).
+How it works:
+) It takes file with postedit triplets (s, mt, pe).
+) If here is '*' in mt, algorithm adds triplet to "potential bidix entries" list.
+) If not, the script calculates the following metric:
+* letters = number of letters in pe
+* distance = Levenshtein distance betweeen mt and pe
+((letters - distance) / letters) * 100
+If 50 <= this number < 100, the algorithm adds triplet to "grammar mistakes" list.
+) Else the algorithm checks, if mt != pe, and if no, adds triplet to "other mistakes" list.
+==== Cleaning ====
+The extracting postedits algorithm is not perfect and extracts a lot of garbage along with potentially good triplets. For cleaning files with postedits a following script was written: https://github.com/deltamachine/naive-automatic-postediting/blob/master/new_alg/clean_postedits.py. On the first step it tags every part of every triplet using apertium-tagger and then drops triplets which contain punctuation. It helps filter out wrong triplets like, for example, (',', '*видець', ',').
+Then it calculates the same metric as in classifying step between s and mt, mt and pe, s and pe. If every result >= 30 and triplet is not from "other mistakes" list, the algorithm saves this triplet, if not - drops it.
+==== Inserting operations into a language pair ====
+==== Evaluation ====