Assimilation Evaluation Toolkit - Revision history

Unhammer: pdf url

2015-04-23T09:04:54Z

pdf url

Ilnar.salimzyan: fix links to eamt-assim paper

2015-03-13T15:06:00Z

fix links to eamt-assim paper

Sereni: /* Experiments */

2014-08-22T19:16:49Z

Experiments

Sereni: /* Progress */

2014-08-22T18:57:27Z

Progress

Sereni: /* Experiments */

2014-08-21T11:17:45Z

Experiments

Sereni: /* Progress */

2014-08-20T19:40:01Z

Progress

Sereni: /* Experiments */

2014-08-20T17:48:20Z

Experiments

Sereni at 19:31, 13 August 2014

2014-08-13T19:31:04Z

Sereni at 17:36, 2 July 2014

2014-07-02T17:36:05Z

Sereni: deleted the how-to-use section

2014-06-27T11:15:30Z

deleted the how-to-use section

@@ Line 20: / Line 20: @@
 == Multiple choice gaps ==
-An additional task generation mode features multiple choice options for gaps. Each omitted word is assigned a list of similar words for the user to choose from during evaluation. The choices are picked from the same text by part of speech and grammar tags. A choice must be the same part of speech as the original word, and they should share as many grammatical features as possible. The approach is described in <ref>Trond Trosterud, Kevin Brubeck Unhammer. Evaluating North Sámi to Norwegian assimilation RBMT. Proceedings of the Third International Workshop on Free/Open-Source Rule-Based Machine Translation (FreeRBMT 2012); 06/2012</ref> and
+An additional task generation mode features multiple choice options for gaps. Each omitted word is assigned a list of similar words for the user to choose from during evaluation. The choices are picked from the same text by part of speech and grammar tags. A choice must be the same part of speech as the original word, and they should share as many grammatical features as possible. The approach is described in <ref>Trond Trosterud, Kevin Brubeck Unhammer. [http://www.molto-project.eu/sites/default/files/FreeRBMT-2012.pdf#19 Evaluating North Sámi to Norwegian assimilation RBMT]. Proceedings of the Third International Workshop on Free/Open-Source Rule-Based Machine Translation (FreeRBMT 2012); 06/2012</ref> and
 <ref>Jim O'Regan and Mikel L. Forcada (2013) "[http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/download/4867/2882 Peeking through the language barrier: the development of a free/open-source gisting system for Basque to English based on apertium.org]". Procesamiento del Lenguaje Natural 51, 15-22.</ref>. An example of keyword choices generated on the two sentences above (the number of choices can be specified):

@@ Line 32: / Line 32: @@
 Four groups of evaluators (a total of 11) are asked to fill the gaps in Spanish sentences with appropriate words. Each group is given 9 sentences with different hints, so that all sentences are evaluated in different modes, but no user evaluates one sentence in two modes at once. The hint modes include the following: no assistance, the source Basque sentence, machine translation of Basque sentence into Spanish using Apertium, and both the source Basque sentence and its machine translation.
-Source data and experiment results can be found in Apertium SVN repository<ref>Basque-Spanish experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eu-es/]</ref>. The results of the evaluation are presented in tables. For tables 2 and 4, the evaluators' answers have been reviewed to determine whether any of the words that did not match the answer key could be used in each question. The list of candidate synonyms has been created automatically based on the provided answers. For a word to be included into the list, it must have been submitted by two or more evaluators. The list has been reviewed by a native speaker of Spanish, and the relevant synonyms have been counted as correct answers in the results.
+Source data and experiment results can be found in Apertium SVN repository<ref>Basque-Spanish experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/2015-eamt-assim/eu-es]</ref>. The results of the evaluation are presented in tables. For tables 2 and 4, the evaluators' answers have been reviewed to determine whether any of the words that did not match the answer key could be used in each question. The list of candidate synonyms has been created automatically based on the provided answers. For a word to be included into the list, it must have been submitted by two or more evaluators. The list has been reviewed by a native speaker of Spanish, and the relevant synonyms have been counted as correct answers in the results.
-A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eng-kaz/]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment. A synonyms list was built with the help of a native speaker of Kazakh.
+A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/2015-eamt-assim/eng-kaz]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment. A synonyms list was built with the help of a native speaker of Kazakh.
-Another experiment was conducted for Tatar-Russian language pair <ref>Tatar-Russian experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/tat-rus/]</ref>. 28 evaluators have participated in it, each filling a set of 36 sentences divided into 3 groups with 10, 20 and 30% of words removed, in 4 different assistance modes, similarly to the Basque-Spanish experiment. The texts for this experiment belong to three domains: casual conversations, legal texts and news extracts. The results were calculated for each set separately, along with the united results for all of them. A synonyms list was also used, built by a native Russian speaker based on evaluators' answers.
+Another experiment was conducted for Tatar-Russian language pair <ref>Tatar-Russian experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/2015-eamt-assim/tat-rus]</ref>. 28 evaluators have participated in it, each filling a set of 36 sentences divided into 3 groups with 10, 20 and 30% of words removed, in 4 different assistance modes, similarly to the Basque-Spanish experiment. The texts for this experiment belong to three domains: casual conversations, legal texts and news extracts. The results were calculated for each set separately, along with the united results for all of them. A synonyms list was also used, built by a native Russian speaker based on evaluators' answers.
 == Progress ==

@@ Line 35: / Line 35: @@
 A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eng-kaz/]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment. A synonyms list was built with the help of a native speaker of Kazakh.
+Another experiment was conducted for Tatar-Russian language pair <ref>Tatar-Russian experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/tat-rus/]</ref>. 28 evaluators have participated in it, each filling a set of 36 sentences divided into 3 groups with 10, 20 and 30% of words removed, in 4 different assistance modes, similarly to the Basque-Spanish experiment. The texts for this experiment belong to three domains: casual conversations, legal texts and news extracts. The results were calculated for each set separately, along with the united results for all of them. A synonyms list was also used, built by a native Russian speaker based on evaluators' answers.
 == Progress ==

@@ Line 59: / Line 59: @@
 '''Week 11''': Added scripts to prepare and distribute the tasks automatically given a number of evaluators, number of sentences and evaluation modes. Prepared material for Basque-Spanish evaluations.
-'''Week 12''': Collected the results of Basque-Spanish evaluations. Prepared material for Kazakh-English evaluations. Collected the results of Kazakh-English evaluations.
+'''Week 12''': Collected the results of Basque-Spanish evaluations. Prepared material for and collected results of English-Kazakh and Tatar-Russian evaluations.
 == References and links ==

@@ Line 34: / Line 34: @@
 Source data and experiment results can be found in Apertium SVN repository<ref>Basque-Spanish experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eu-es/]</ref>. The results of the evaluation are presented in tables. For tables 2 and 4, the evaluators' answers have been reviewed to determine whether any of the words that did not match the answer key could be used in each question. The list of candidate synonyms has been created automatically based on the provided answers. For a word to be included into the list, it must have been submitted by two or more evaluators. The list has been reviewed by a native speaker of Spanish, and the relevant synonyms have been counted as correct answers in the results.
-A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eng-kaz/]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment.
+A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eng-kaz/]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment. A synonyms list was built with the help of a native speaker of Kazakh.
 == Progress ==

@@ Line 26: / Line 26: @@
 == Experiments ==
+As a part of the testing process, the toolkit has been used to evaluate Basque-Spanish and English-Kazakh Apertium language pairs.
-As a part of the testing process, the toolkit has been used to evaluate Basque-Spanish Apertium language pair. The experiment features 36 sentence pairs over 10 words long randomly drawn from the parallel corpus <ref>Memorias de traducción del Servicio Oficial de Traductores del IVAP [http://opendata.euskadi.net/w79-contdata/es/contenidos/ds_recursos_linguisticos/memorias_traduccion/es_izo/memorias_traduccion_izo.html]</ref> of legal texts. The sentences are divided into three groups, and 10, 20 and 30 percent of words are removed from Spanish sentences in each group respectively. The removed words belong to the following parts of speech: noun, proper noun, adverb, adjective and lexical verb.
+The Basque-Spanish experiment features 36 sentence pairs over 10 words long randomly drawn from the parallel corpus <ref>Memorias de traducción del Servicio Oficial de Traductores del IVAP [http://opendata.euskadi.net/w79-contdata/es/contenidos/ds_recursos_linguisticos/memorias_traduccion/es_izo/memorias_traduccion_izo.html]</ref> of legal texts. The sentences are divided into three groups, and 10, 20 and 30 percent of words are removed from Spanish sentences in each group respectively. The removed words belong to the following parts of speech: noun, proper noun, adverb, adjective and lexical verb.
 Four groups of evaluators (a total of 11) are asked to fill the gaps in Spanish sentences with appropriate words. Each group is given 9 sentences with different hints, so that all sentences are evaluated in different modes, but no user evaluates one sentence in two modes at once. The hint modes include the following: no assistance, the source Basque sentence, machine translation of Basque sentence into Spanish using Apertium, and both the source Basque sentence and its machine translation.
-Source data and experiment results can be found on Github<ref>Basque-Spanish experiment data [https://github.com/Sereni/Appraise/tree/master/eval/Basque-Spanish%20evaluation]</ref>. The results of the evaluation are presented in tables. For tables 2 and 4, the evaluators' answers have been reviewed to determine whether any of the words that did not match the answer key could be used in each question. The list of candidate synonyms has been created automatically based on the provided answers. For a word to be included into the list, it must have been submitted by two or more evaluators. The list has been reviewed by a native speaker of Spanish, and the relevant synonyms have been counted as correct answers in the results.
+Source data and experiment results can be found in Apertium SVN repository<ref>Basque-Spanish experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eu-es/]</ref>. The results of the evaluation are presented in tables. For tables 2 and 4, the evaluators' answers have been reviewed to determine whether any of the words that did not match the answer key could be used in each question. The list of candidate synonyms has been created automatically based on the provided answers. For a word to be included into the list, it must have been submitted by two or more evaluators. The list has been reviewed by a native speaker of Spanish, and the relevant synonyms have been counted as correct answers in the results.
+A similar set of data is provided for English-Kazakh evaluations <ref>English-Kazakh experiment data [https://svn.code.sf.net/p/apertium/svn/branches/papers/assim-eamt-2015/eng-kaz/]</ref>. 9 Kazakh speakers completed the evaluation of 36 sentences in 4 different modes analogous to Basque-Spanish experiment.
 == Progress ==

@@ Line 1: / Line 1: @@
 == Project description ==
-This page describes a work in progress<ref>Current code [https://github.com/Sereni/Appraise/tree/master/eval on github]</ref>. The [[Assimilation and Dissemination|Assimilation]] evaluation toolkit is a set of programs that generates tasks for human evaluation of machine translation. The tasks consist of sentences in the original language, reference translation with keywords omitted and the machine translation of these sentences. They also contain a key to determine answer correctness, not shown to the evaluator.
+This page describes a work in progress<ref>Current code [https://github.com/Sereni/Appraise/tree/master/eval on github]</ref>. The [[Assimilation and Dissemination|Assimilation]] evaluation toolkit is a set of programs that generates tasks for human evaluation of machine translation. The tasks consist of sentences in the original language, reference translation with keywords omitted and the machine translation of these sentences. They also contain a key to determine answer correctness, not shown to the evaluator. The tasks may be generated as standalone text files with automated checking, or as XML files to be integrated into [https://github.com/cfedermann/Appraise Appraise] evaluation system.
 == Keyword extraction ==
@@ Line 37: / Line 37: @@
 '''Week 5''': Created the command-line interface for task generation and answer checking.
+'''Weeks 6-7''': Integrated gisting evaluation tasks into Appraise. Created command-line interface for XML generation.
 == References and links ==

@@ Line 1: / Line 1: @@
 == Project description ==
 This page describes a work in progress<ref>Current code [https://github.com/Sereni/Appraise/tree/master/eval on github]</ref>. The [[Assimilation and Dissemination|Assimilation]] evaluation toolkit is a set of programs that generates tasks for human evaluation of machine translation. The tasks consist of sentences in the original language, reference translation with keywords omitted and the machine translation of these sentences. They also contain a key to determine answer correctness, not shown to the evaluator.
-== How to use ==
-To use the toolkit, download the files from [https://github.com/Sereni/Appraise/tree/master/eval github]. The Python files are required, and you may either use the default text files for testing or provide your own data. The toolkit has one dependency, click, which can be installed from pip:
- $ pip install click
-To generate tasks, run
- $ python gist_eval.py [OPTIONS] ORIGINAL REFERENCE TAGS [TASK] [KEYS]
-ORIGINAL is the path to untranslated text, REFERENCE - to reference translation, from which the gapped text will be created, TAGS - the reference translation put through Apertium tagger. TASK and KEYS are optional arguments, which specify the path to save output files - the task itself and the answer keys, respectively. If those are left out, the defaults will be task.txt and keys.txt in the script folder.
-The following options are supported:
-* -mt, --machine FILENAME – original text translated through Apertium, if the tasks should contain machine translation as a tip for evaluator;
-* -m, --mode [simple | choices | lemmas] - specifies task mode. Simple just removes the words, choices gives a choice of three options for each gap, and lemmas leaves the word's lemma in place, and the user is to fill in the correct grammatical form. The default mode is 'simple';
-* -k, --keyword - if on, the words to be removed will be determined with keyword selection algorithm, if off - the words will be randomly selected;
-* -d, --density - an integer from 1 to 100, specifies gap density. The default is 50;
-* -r, --relative - if on, the gap density is calculated against the number of words which were selected to be removed, if off - against the total number of words in the text (but not more than the number of keywords found, if the keyword mode is on);
-* -p, --pos - if you wish to remove only specific parts of speech, specify them here as a string of [http://wiki.apertium.org/wiki/List_of_symbols Apertium part-of-speech tags] separated by commas, i.e. 'vblex, n, adj'.
-To check the task completed by the user, run
- $ python text_checker.py TASK KEYS
-TASK is a path to the filled-in task, and KEYS is a path to the answer keys, which were generated with the task. Note that the script assumes that the filled-in task structure is unchanged, and that the evaluator only filled in / changed the words in the gaps. The script returns the number and percentage of correct answers based on the answer key.
 == Keyword extraction ==