Evalution kaz-tur Machine Translation System - Revision history

Purplemoon at 10:16, 24 October 2018

2018-10-24T10:16:46Z

Purplemoon at 19:49, 23 October 2018

2018-10-23T19:49:58Z

Purplemoon at 18:54, 21 October 2018

2018-10-21T18:54:37Z

Purplemoon at 18:21, 21 October 2018

2018-10-21T18:21:14Z

Purplemoon at 16:26, 17 September 2018

2018-09-17T16:26:29Z

Purplemoon at 16:19, 17 September 2018

2018-09-17T16:19:14Z

Purplemoon at 16:18, 17 September 2018

2018-09-17T16:18:26Z

Purplemoon at 15:52, 17 September 2018

2018-09-17T15:52:40Z

Purplemoon at 15:51, 17 September 2018

2018-09-17T15:51:34Z

Purplemoon at 15:42, 17 September 2018

2018-09-17T15:42:38Z

@@ Line 21: / Line 21: @@
 The text files which used to calculate the evaluation was putted on github https://github.com/apertium/apertium-kaz-tur/tree/master/eval.
-Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower. The differential evaluation results add into apertium-kaz-tur in github https://github.com/apertium/apertium-kaz-tur/blob/master/eval/test2.txt
+Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower. The differential evaluation results add into apertium-kaz-tur in github https://github.com/apertium/apertium-kaz-tur/blob/master/eval/test.txt
-We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium's structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both output.
+We manually checked each of the translation output by the structural-transfer module to see if the applying rules with new apertium were better or worse than the old apertium. We found out the new apertium in 33 sentences out of 100 was better than old apertium, just in 3 sentences out of 100 the old apertium achive better than the new apertium. Also, in 56 out of 100 sentences both systems got the right translation, and in 7 out of 100 both of them had bad translation.

@@ Line 21: / Line 21: @@
 The text files which used to calculate the evaluation was putted on github https://github.com/apertium/apertium-kaz-tur/tree/master/eval.
-Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.
+Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower. The differential evaluation results add into apertium-kaz-tur in github https://github.com/apertium/apertium-kaz-tur/blob/master/eval/test2.txt
 We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium's structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both output.

@@ Line 18: / Line 18: @@
 Besides calculating WER and PER for our new apertium Kazakh-Turkish MT system, we did the same for old apertium Kazakh-Turkish MT system. The procedure was the same for both of them. We took a small (1,025 tokens) Kazakh text, which was a concatenation of several articles from Wikipedia and translated it using the two MT systems. The output of each system was postedited independently to avoid biasing in favour of one particular system.
 Then we calculated WER and PER for each using the apertium-eval-translator {http://wiki.apertium.org/wiki/apertium-eval-translator}.
+The text files which used to calculate the evaluation was putted on github https://github.com/apertium/apertium-kaz-tur/tree/master/eval.
 Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.

@@ Line 8: / Line 8: @@
 |-
 |new-system
-|23.57
+|20.87
-|23.19
+|19.98
 |-
 |old-system

← Older revision		Revision as of 16:26, 17 September 2018
Line 21:		Line 21:
	Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.		Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.

	We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium's structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both ~~out~~.		We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium's structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both output.

← Older revision		Revision as of 16:19, 17 September 2018
Line 21:		Line 21:
	Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.		Because the error rate was low for the new-apertium, we chose to do a differential evaluation compared to the old-apertium, to see if in fact, the error rate was actually lower.

	We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium' structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both out.		We manually checked each of the translations output by the structural-transfer module to see if the applying rules were better or worse than the old apertium's structural-transfer module. We found out that the new module's rate was %93 and the old module's rate was %73 for both out of 100 sentences, there are %20 differences between two systems and about %7 sentences were bad in both out.

← Older revision		Revision as of 15:52, 17 September 2018
Line 17:		Line 17:

	Besides calculating WER and PER for our Kazakh-Turkish MT system, we did the same for old apertium Kazakh-Turkish MT system. The procedure was the same for both of them. We took a small (1,025 tokens) Kazakh text, which was a concatenation of several articles from Wikipedia and translated it using the two MT systems. The output of each system was postedited independently to avoid biasing in favour of one particular system.		Besides calculating WER and PER for our Kazakh-Turkish MT system, we did the same for old apertium Kazakh-Turkish MT system. The procedure was the same for both of them. We took a small (1,025 tokens) Kazakh text, which was a concatenation of several articles from Wikipedia and translated it using the two MT systems. The output of each system was postedited independently to avoid biasing in favour of one particular system.
	Then we calculated WER and PER for each using the apertium-eval-translator ~~tool\footnote{\url~~{http://wiki.apertium.org/wiki/apertium-eval-translator}}		Then we calculated WER and PER for each using the apertium-eval-translator {http://wiki.apertium.org/wiki/apertium-eval-translator}.

@@ Line 15: / Line 15: @@
 |41.69
 |}
+Besides calculating WER and PER for our Kazakh-Turkish MT system, we did the same for old apertium Kazakh-Turkish MT system. The procedure was the same for both of them. We took a small (1,025 tokens) Kazakh text, which was a concatenation of several articles from Wikipedia and translated it using the two MT systems. The output of each system was postedited independently to avoid biasing in favour of one particular system.
+Then we calculated WER and PER for each using the apertium-eval-translator tool\footnote{\url{http://wiki.apertium.org/wiki/apertium-eval-translator}}

@@ Line 5: / Line 5: @@
 {|class="wikitable"
 |-
-! system !! WER(%) !! PER(%)
+! System !! WER(%) !! PER(%)
 |-
 |new-system