Difference between revisions of "Evaluation"
Jump to navigation
Jump to search
Line 1: | Line 1: | ||
New section coming soon! |
New section coming soon! |
||
+ | |||
+ | ==Using apertium-eval-translator for WER and PER== |
||
+ | |||
+ | <code>apertium-eval-translator</code> is a script written in Perl. It calculates the word error rate (WER) and the position-independent word error rate (PER) between a translation performed by an Apertium-based MT system and its human-corrected translation at document level. Although it has been designed to evaluate Apertium-based systems, it can be easily adapted to evaluate other MT systems. |
||
+ | |||
+ | ===Usage=== |
||
+ | <pre> |
||
+ | apertium-eval-translator -test testfile -ref reffile [-beam <n>] |
||
+ | |||
+ | Options: |
||
+ | |||
+ | -test|-t Specify the file with the translation to evaluate |
||
+ | -ref|-r Specify the file with the reference translation |
||
+ | -beam|-b Perform a beam search by looking only to the <n> previous |
||
+ | and <n> posterior neigboring words (optional parameter |
||
+ | to make the evaluation much faster) |
||
+ | -help|-h Show this help message |
||
+ | -version|-v Show version information and exit |
||
+ | |||
+ | Note: The <n> value provided with -beam is language-pair dependent. The |
||
+ | closer the languages involved are, the lesser <n> can be without |
||
+ | affecting the evaluation results. This parameter only affects the WER |
||
+ | evaluation. |
||
+ | |||
+ | Note: Reference translation MUST have no unknown-word marks, even if |
||
+ | they are free rides. |
||
+ | |||
+ | This software calculates (at document level) the word error rate (WER) |
||
+ | and the postion-independent word error rate (PER) between a translation |
||
+ | performed by the Apertium MT system and a reference translation obtained |
||
+ | by post-editing the system ouput. |
||
+ | |||
+ | It is assumed that unknow words are marked with a start (*), as Apertium |
||
+ | does; nevertheless, it can be easily adapted to evaluate other MT |
||
+ | systems that do not mark unknown words with a star. |
||
+ | </pre> |
||
+ | |||
+ | See [[English and Esperanto/Evaluation]] for an example. |
||
+ | |||
==Evaluating with Wikipedia== |
==Evaluating with Wikipedia== |
Revision as of 19:27, 17 September 2009
New section coming soon!
Using apertium-eval-translator for WER and PER
apertium-eval-translator
is a script written in Perl. It calculates the word error rate (WER) and the position-independent word error rate (PER) between a translation performed by an Apertium-based MT system and its human-corrected translation at document level. Although it has been designed to evaluate Apertium-based systems, it can be easily adapted to evaluate other MT systems.
Usage
apertium-eval-translator -test testfile -ref reffile [-beam <n>] Options: -test|-t Specify the file with the translation to evaluate -ref|-r Specify the file with the reference translation -beam|-b Perform a beam search by looking only to the <n> previous and <n> posterior neigboring words (optional parameter to make the evaluation much faster) -help|-h Show this help message -version|-v Show version information and exit Note: The <n> value provided with -beam is language-pair dependent. The closer the languages involved are, the lesser <n> can be without affecting the evaluation results. This parameter only affects the WER evaluation. Note: Reference translation MUST have no unknown-word marks, even if they are free rides. This software calculates (at document level) the word error rate (WER) and the postion-independent word error rate (PER) between a translation performed by the Apertium MT system and a reference translation obtained by post-editing the system ouput. It is assumed that unknow words are marked with a start (*), as Apertium does; nevertheless, it can be easily adapted to evaluate other MT systems that do not mark unknown words with a star.
See English and Esperanto/Evaluation for an example.
Evaluating with Wikipedia
- Main article: Evaluating with Wikipedia