Kazakh and Tatar/Work plan - Revision history

Ilnar.salimzyan at 19:47, 8 May 2014

2014-05-08T19:47:12Z

Ilnar.salimzyan at 13:22, 16 March 2014

2014-03-16T13:22:05Z

Ilnar.salimzyan: Minor cleanup

2014-02-05T10:28:22Z

Minor cleanup

Francis Tyers at 15:28, 8 March 2013

2013-03-08T15:28:53Z

Ilnar.salimzyan: add link to GSoC final evaluation page

2013-01-30T10:04:42Z

add link to GSoC final evaluation page

Ilnar.salimzyan at 13:34, 6 August 2012

2012-08-06T13:34:34Z

Ilnar.salimzyan at 11:20, 6 August 2012

2012-08-06T11:20:10Z

Ilnar.salimzyan at 22:58, 31 July 2012

2012-07-31T22:58:52Z

Ilnar.salimzyan at 17:59, 23 July 2012

2012-07-23T17:59:28Z

Ilnar.salimzyan at 16:58, 16 July 2012

2012-07-16T16:58:38Z

@@ Line 35: / Line 35: @@
 | 11      || <s>30/07&mdash;05/08</s>  ||   80%      || 79.9%,&nbsp;77.4%  ||  - || || -   || ||  Work on disambiguation.
 |-
-| 12      || <s>06/08&mdash;12/08</s>  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''[http://wiki.apertium.org/wiki/Kazakh_and_Tatar/Evaluation Final evaluation]'''. Tidying up, releasing
+| 12      || <s>06/08&mdash;12/08</s>  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''Final evaluation''' (see below). Tidying up, releasing
 |-
 |}
+== Final evaluation ==
+Sources of texts can be find here: [https://svn.code.sf.net/p/apertium/svn/trunk/apertium-kaz-tat/texts/README README]
+=== Azattyq news ===
+<pre>
+selimcan@achilles:~/src/apertium/incubator/apertium-kaz-tat$ perl ../../trunk/apertium-eval-translator/apertium-eval-translator.pl -test texts/4story.kaz-tat.02_09_2012.txt -ref texts/4story.kaz-tat.postedit.txt
+Test file: 'texts/4story.kaz-tat.02_09_2012.txt'
+Reference file 'texts/4story.kaz-tat.postedit.txt'
+Statistics about input files
+-------------------------------------------------------
+Number of words in reference: 534
+Number of words in test: 536
+Number of unknown words (marked with a star) in test: 2
+Percentage of unknown words: 0.37 %
+Results when removing unknown-word marks (stars)
+-------------------------------------------------------
+Edit distance: 133
+Word error rate (WER): 24.91 %
+Number of position-independent correct words: 413
+Position-independent word error rate (PER): 23.03 %
+Results when unknown-word marks (stars) are not removed
+-------------------------------------------------------
+Edit distance: 135
+Word Error Rate (WER): 25.28 %
+Number of position-independent correct words: 411
+Position-independent word error rate (PER): 23.41 %
+Statistics about the translation of unknown words
+-------------------------------------------------------
+Number of unknown words which were free rides: 2
+Percentage of unknown words that were free rides: 100.00 %
+</pre>
+=== Wikipedia article ===
+<pre>
+selimcan@achilles:~/src/apertium/incubator/apertium-kaz-tat$ perl ../../trunk/apertium-eval-translator/apertium-eval-translator.pl -test texts/5story.kaz-tat.01_09_2012.txt -ref texts/5story.kaz-tat.postedit.txt
+Test file: 'texts/5story.kaz-tat.01_09_2012.txt'
+Reference file 'texts/5story.kaz-tat.postedit.txt'
+Statistics about input files
+-------------------------------------------------------
+Number of words in reference: 515
+Number of words in test: 518
+Number of unknown words (marked with a star) in test:
+Percentage of unknown words: 0.00 %
+Results when removing unknown-word marks (stars)
+-------------------------------------------------------
+Edit distance: 61
+Word error rate (WER): 11.84 %
+Number of position-independent correct words: 460
+Position-independent word error rate (PER): 11.26 %
+Results when unknown-word marks (stars) are not removed
+-------------------------------------------------------
+Edit distance: 61
+Word Error Rate (WER): 11.84 %
+Number of position-independent correct words: 460
+Position-independent word error rate (PER): 11.26 %
+Statistics about the translation of unknown words
+-------------------------------------------------------
+Number of unknown words which were free rides: 0
+Percentage of unknown words that were free rides: 0%
+</pre>
 == Notes ==

← Older revision		Revision as of 13:22, 16 March 2014
Line 1:		Line 1:
	This is a workplan for development efforts for [[Kazakh and Tatar]] translator in [[Google Summer of Code]] 2012.		This was a workplan for development efforts for [[Kazakh and Tatar]] translator in [[Google Summer of Code]] 2012.

	* Trimmed coverage means the coverage of both of the morphological analysers after being trimmed according to the bilingual dictionary of the pair, that is, only containing stems which are also in the bilingual dictionary.		* Trimmed coverage means the coverage of both of the morphological analysers after being trimmed according to the bilingual dictionary of the pair, that is, only containing stems which are also in the bilingual dictionary.

@@ Line 5: / Line 5: @@
 * Evaluation is taking <math>n</math> words and performing an [[evaluation]] for post-edition word error rate (WER). The output for those <math>n</math> words should be clean.
-("news" is <code>tat.news.2005-2011_300K-sentences.txt.bz2</code>, "wp" - <code>kaz.fixed.wikipedia.2010_100k-sentences.txt.bz2</code> <ref>Despite the fact that both lexicons have the same amount of stems, transducer for Tatar usually has a  lower coverage. This is because some forms aren't implemented yet, e.g. some verbal forms. Here I gave numbers for both directions, although the domains are different. Not too sure about it, hope this is ok.</ref>)
+("news" is <code>tat.news.2005-2011_300K-sentences.txt.bz2</code>, "wp" - <code>kaz.fixed.wikipedia.2010_100k-sentences.txt.bz2</code> <ref>Despite the fact that both lexicons have the same amount of stems, transducer for Tatar usually has a  lower coverage. This is probably because some forms aren't implemented yet, e.g. some verbal forms. Here I give numbers for both directions. Note that domains are different.</ref>)
 {|class=wikitable
@@ Line 35: / Line 35: @@
 | 11      || <s>30/07&mdash;05/08</s>  ||   80%      || 79.9%,&nbsp;77.4%  ||  - || || -   || ||  Work on disambiguation.
 |-
-| 12      || 06/08&mdash;12/08  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''[http://wiki.apertium.org/wiki/Kazakh_and_Tatar/Evaluation Final evaluation]'''. Tidying up, releasing
+| 12      || <s>06/08&mdash;12/08</s>  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''[http://wiki.apertium.org/wiki/Kazakh_and_Tatar/Evaluation Final evaluation]'''. Tidying up, releasing
 |-
 |}

@@ Line 43: / Line 43: @@
 [[Category:Kazakh and Tatar]]
-[[Category:Google Summer of Code]]
 [[Category:Turkic languages]]

@@ Line 35: / Line 35: @@
 | 11      || <s>30/07&mdash;05/08</s>  ||   80%      || 79.9%,&nbsp;77.4%  ||  - || || -   || ||  Work on disambiguation.
 |-
-| 12      || 06/08&mdash;12/08  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''Final evaluation'''. Tidying up, releasing
+| 12      || 06/08&mdash;12/08  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''[http://wiki.apertium.org/wiki/Kazakh_and_Tatar/Evaluation Final evaluation]'''. Tidying up, releasing
 |-
 |}

@@ Line 31: / Line 31: @@
 | 9       || <s>16/07&mdash;22/07</s>  ||   77%        || 77.7%,&nbsp;75.5%  ||  - || || 200 words ||  || Work on disambiguation.
 |-
-| 10      || <s>23/07&mdash;29/07</s>  ||   80%         || 79.1%,&nbsp;76.8 %   ||  {{tag|v}} || No || -  || || Verbs should be testvoc clean.
+| 10      || <s>23/07&mdash;29/07</s>  ||   80%         || 79.1%,&nbsp;76.8%   ||  {{tag|v}} || No || -  || || Verbs should be testvoc clean.
 |-
-| 11      || <s>30/07&mdash;05/08</s>  ||   80%      || 79.9%,  ||  - || || -   || ||  Work on disambiguation.
+| 11      || <s>30/07&mdash;05/08</s>  ||   80%      || 79.9%,&nbsp;77.4%  ||  - || || -   || ||  Work on disambiguation.
 |-
 | 12      || 06/08&mdash;12/08  ||   80%        ||    || ''all categories clean'' ||  || 500 words  || || '''Final evaluation'''. Tidying up, releasing

@@ Line 27: / Line 27: @@
 | 7       || <s>02/07&mdash;08/07</s>  ||   73%       ||  73.5%,&nbsp;70.4% ||  || || -   || ||  Worked on translation of kaz.crp.txt
 |-
-| 8       || <s>09/07&mdash;15/07</s>  ||   75%         || 76.2,%&nbsp;72.7% || {{tag|prn}} || No || -   || || Pronouns should be testvoc clean.
+| 8       || <s>09/07&mdash;15/07</s>  ||   75%         || 76.2%,&nbsp;72.7% || {{tag|prn}} || No || -   || || Pronouns should be testvoc clean.
 |-
-| 9       || <s>16/07&mdash;22/07</s>  ||   77%        || 77.7,%&nbsp;75.5%  ||  - || || 200 words ||  || Work on disambiguation.
+| 9       || <s>16/07&mdash;22/07</s>  ||   77%        || 77.7%,&nbsp;75.5%  ||  - || || 200 words ||  || Work on disambiguation.
 |-
-| 10      || 23/07&mdash;29/07  ||   80%         ||   ||  {{tag|v}} || || -  || || Verbs should be testvoc clean.
+| 10      || <s>23/07&mdash;29/07</s>  ||   80%         || 79.1%,&nbsp;76.8 %   ||  {{tag|v}} || No || -  || || Verbs should be testvoc clean.
 |-
 | 11      || 30/07&mdash;05/08  ||   80%      ||      ||  - || || -   || ||  Work on disambiguation.

@@ Line 25: / Line 25: @@
 | 6       || <s>25/06&mdash;01/07</s>  ||   70%        || 73.4%,&nbsp;70.3%   ||  {{tag|n}} {{tag|num><subst}} {{tag|np}} {{tag|adj><subst}} || ||  500 words   || 23.79%<ref>apertium-kaz-tat$ perl ../../trunk/apertium-eval-translator/apertium-eval-translator.pl -t texts/3story.kaz-tat.12072012.txt -r texts/3story.kaz-tat.postedit.txt</ref>||  '''Midterm evaluation'''. Work on disambiguation.
 |-
-| 7       || <s>02/07&mdash;08/07</s>  ||   73%       ||  73.5%,&nbsp;70.4%   ||  || || -   || ||  Worked on translation of kaz.crp.txt
+| 7       || <s>02/07&mdash;08/07</s>  ||   73%       ||  73.5%,&nbsp;70.4% ||  || || -   || ||  Worked on translation of kaz.crp.txt
 |-
-| 8       || 09/07&mdash;15/07  ||   75%         ||   || {{tag|prn}} || || -   || || Pronouns should be testvoc clean.
+| 8       || <s>09/07&mdash;15/07</s>  ||   75%         || 76.2,%&nbsp;72.7% || {{tag|prn}} || No || -   || || Pronouns should be testvoc clean.
 |-
 | 9       || 16/07&mdash;22/07  ||   77%        ||    ||  - || || 200 words ||  || Work on disambiguation.