Maltese and Arabic/Work plan - Revision history

Francis Tyers at 15:28, 8 March 2013

2013-03-08T15:28:43Z

Mfronczak at 05:32, 28 August 2012

2012-08-28T05:32:25Z

Mfronczak at 08:51, 15 August 2012

2012-08-15T08:51:53Z

Mfronczak at 10:49, 8 August 2012

2012-08-08T10:49:54Z

Unhammer at 09:58, 6 August 2012

2012-08-06T09:58:23Z

Unhammer: note on WER interpreting

2012-08-06T09:57:32Z

note on WER interpreting

Unhammer: hide it a bit

2012-08-06T09:55:30Z

hide it a bit

Unhammer: Google WER was a real post-edit too

2012-08-06T09:54:36Z

Google WER was a real post-edit too

Mfronczak at 08:37, 31 July 2012

2012-07-31T08:37:53Z

Mfronczak at 12:00, 24 July 2012

2012-07-24T12:00:40Z

← Older revision		Revision as of 15:28, 8 March 2013
Line 40:		Line 40:

	[[Category:Maltese and Arabic]]		[[Category:Maltese and Arabic]]
	[[Category:Google Summer of Code]]

@@ Line 23: / Line 23: @@
 | 5       || <s>18/06&mdash;24/06</s>  ||   68%           || 79.0%, 78.3% ||  {{tag|adj}} || -   || - || - || Bidix expansion; bidix adjectives should be testvoc clean.
 |-
-| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11% || (33.53 %) || '''Midterm evaluation'''. Bidix expansion, multiwords.
+| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11 % || (33.53 %) || '''Midterm evaluation'''. Bidix expansion, multiwords.
 |-
 | 7       || <s>02/07&mdash;08/07</s>  ||   73%           || 79.4%, 78.6% ||  - || -   || - || - || Transfer
@@ Line 29: / Line 29: @@
 | 8       || <s>09/07&mdash;15/07</s>  ||   75%           || 79.5%, 78.8% ||  - || -   || - || - || Transfer, corpus testvoc
 |-
-| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28% || (35.89 %) || Disambiguation, retrain tagger
+| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28 % || (35.89 %) || Disambiguation, retrain tagger
 |-
 | 10      || <s>23/07&mdash;29/07</s>  ||   80%           || 80.9%, 80.2% ||  {{tag|v}} || -  || - || - || Testvoc
@@ Line 35: / Line 35: @@
 | 11      || <s>30/07&mdash;05/08</s>  ||   80%           || 81.2%, 80.5%  || - || -  || - || - || Testvoc
 |-
-| 12      || <s>06/08&mdash;12/08</s>  ||   80%           || 82.0%, 80.9% || ''all categories clean'' || 500 words  || ? || - || '''Final evaluation'''. Tidying up, releasing
+| 12      || <s>06/08&mdash;12/08</s>  ||   80%           || 82.0%, 80.9% || ''all categories clean'' || 500 words  || 21.34 % || (39.00 %) || '''Final evaluation'''. Tidying up, releasing
 |-
 |}

@@ Line 35: / Line 35: @@
 | 11      || <s>30/07&mdash;05/08</s>  ||   80%           || 81.2%, 80.5%  || - || -  || - || - || Testvoc
 |-
-| 12      || 06/08&mdash;12/08  ||   80%           || - || ''all categories clean'' || 500 words  || ? || - || '''Final evaluation'''. Tidying up, releasing
+| 12      || <s>06/08&mdash;12/08</s>  ||   80%           || 82.0%, 80.9% || ''all categories clean'' || 500 words  || ? || - || '''Final evaluation'''. Tidying up, releasing
 |-
 |}

@@ Line 33: / Line 33: @@
 | 10      || <s>23/07&mdash;29/07</s>  ||   80%           || 80.9%, 80.2% ||  {{tag|v}} || -  || - || - || Testvoc
 |-
-| 11      || 30/07&mdash;05/08  ||   80%           || - || - || -  || - || - || Testvoc
+| 11      || <s>30/07&mdash;05/08</s>  ||   80%           || 81.2%, 80.5%  || - || -  || - || - || Testvoc
 |-
 | 12      || 06/08&mdash;12/08  ||   80%           || - || ''all categories clean'' || 500 words  || ? || - || '''Final evaluation'''. Tidying up, releasing

@@ Line 4: / Line 4: @@
 * Testvoc for a category means that the category is [[testvoc]] clean, in both translation directions.
 * Evaluation is taking <math>n</math> words and performing an [[evaluation]] for post-edition word error rate (WER). The output for those <math>n</math> words should be clean, ie. no #/@, and no (or as few as possible) *, for all WER tests.
-** Note on the WER results: both tests use references that are post-edited from the respective MT outputs, but the texts were used in apertium development, so unrestricted texts would probably look quite different.
+** Note on the WER results: both tests use references that are post-edited from the respective MT outputs, but the texts were used in apertium development, so unrestricted texts would probably look quite different. Texts are in SVN in apertium-mt-ar/dev/story/

@@ Line 10: / Line 10: @@
 ! Week    !! Dates              !! B-trimmed Cov. Goal !! B-trimmed Cov. Reached (news, wp) !! Testvoc      !! Evaluation !! WER Reached !! (Google WER) !! Notes
 |-
-| 0       || <s>23/04&mdash;20/05</s>  ||   45%           || 68.6%, 68.1% ||  {{tag|sent}} {{tag|cm}}  {{tag|ij}}  ||   500 words || 3.17 % || 47.44 % || '''Preliminary evaluation'''. Translate the story total coverage and without diagnostics. Get a baseline WER. Aim to get the two mt verb scripts merged.
+| 0       || <s>23/04&mdash;20/05</s>  ||   45%           || 68.6%, 68.1% ||  {{tag|sent}} {{tag|cm}}  {{tag|ij}}  ||   500 words || 3.17 % || (47.44 %) || '''Preliminary evaluation'''. Translate the story total coverage and without diagnostics. Get a baseline WER. Aim to get the two mt verb scripts merged.
 |-
 | 1       || <s>21/05&mdash;27/05</s>  ||   50%           || 74.2%, 73.8% ||  {{tag|num}} {{tag|pr}} || -     || - || - || Basic numerals, and prepositions should be testvoc clean. Maltese verb scripts merged.
@@ Line 16: / Line 16: @@
 | 2       || <s>28/05&mdash;03/06</s>  ||   53%           || 75.3%, 74.5% ||  {{tag|cnjcoo}} {{tag|cnjadv}} {{tag|cnjsub}} || -     || - || - || Create ar verb script based on mt, add ar open-class words according to frequency
 |-
-| 3       || <s>04/06&mdash;10/06</s>  ||   59%           || 75.3%, 74.4% ||  {{tag|adv}} || 200 words    || 8.70 % || 20.94 % || Continue adding open class words to ar (adding to bidix/mt as you go)
+| 3       || <s>04/06&mdash;10/06</s>  ||   59%           || 75.3%, 74.4% ||  {{tag|adv}} || 200 words    || 8.70 % || (20.94 %) || Continue adding open class words to ar (adding to bidix/mt as you go)
 |-
 | 4       || <s>11/06&mdash;17/06</s>  ||   63%           || 76.0%, 75.1% ||  {{tag|prn}} {{tag|det}} || -   || - || - || Monodix completion; closed classes (apart from verb-attached pronouns) should be testvoc clean.
@@ Line 22: / Line 22: @@
 | 5       || <s>18/06&mdash;24/06</s>  ||   68%           || 79.0%, 78.3% ||  {{tag|adj}} || -   || - || - || Bidix expansion; bidix adjectives should be testvoc clean.
 |-
-| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11% || 33.53 % || '''Midterm evaluation'''. Bidix expansion, multiwords.
+| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11% || (33.53 %) || '''Midterm evaluation'''. Bidix expansion, multiwords.
 |-
 | 7       || <s>02/07&mdash;08/07</s>  ||   73%           || 79.4%, 78.6% ||  - || -   || - || - || Transfer
@@ Line 28: / Line 28: @@
 | 8       || <s>09/07&mdash;15/07</s>  ||   75%           || 79.5%, 78.8% ||  - || -   || - || - || Transfer, corpus testvoc
 |-
-| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28% || 35.89 % || Disambiguation, retrain tagger
+| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28% || (35.89 %) || Disambiguation, retrain tagger
 |-
 | 10      || <s>23/07&mdash;29/07</s>  ||   80%           || 80.9%, 80.2% ||  {{tag|v}} || -  || - || - || Testvoc

@@ Line 8: / Line 8: @@
 {|class=wikitable
 |-
-! Week    !! Dates              !! B-trimmed Cov. Goal !! B-trimmed Cov. Reached (news, wp) !! Testvoc      !! Evaluation !! WER Reached !! Notes
+! Week    !! Dates              !! B-trimmed Cov. Goal !! B-trimmed Cov. Reached (news, wp) !! Testvoc      !! Evaluation !! WER Reached !! (Google WER) !! Notes
 |-
-| 0       || <s>23/04&mdash;20/05</s>  ||   45%           || 68.6%, 68.1% ||  {{tag|sent}} {{tag|cm}}  {{tag|ij}}  ||   500 words || 3.17 % || '''Preliminary evaluation'''. Translate the story total coverage and without diagnostics. Get a baseline WER. Aim to get the two mt verb scripts merged.
+| 0       || <s>23/04&mdash;20/05</s>  ||   45%           || 68.6%, 68.1% ||  {{tag|sent}} {{tag|cm}}  {{tag|ij}}  ||   500 words || 3.17 % || 47.44 % || '''Preliminary evaluation'''. Translate the story total coverage and without diagnostics. Get a baseline WER. Aim to get the two mt verb scripts merged.
 |-
-| 1       || <s>21/05&mdash;27/05</s>  ||   50%           || 74.2%, 73.8% ||  {{tag|num}} {{tag|pr}} || -     || - || Basic numerals, and prepositions should be testvoc clean. Maltese verb scripts merged.
+| 1       || <s>21/05&mdash;27/05</s>  ||   50%           || 74.2%, 73.8% ||  {{tag|num}} {{tag|pr}} || -     || - || - || Basic numerals, and prepositions should be testvoc clean. Maltese verb scripts merged.
 |-
-| 2       || <s>28/05&mdash;03/06</s>  ||   53%           || 75.3%, 74.5% ||  {{tag|cnjcoo}} {{tag|cnjadv}} {{tag|cnjsub}} || -     || - || Create ar verb script based on mt, add ar open-class words according to frequency
+| 2       || <s>28/05&mdash;03/06</s>  ||   53%           || 75.3%, 74.5% ||  {{tag|cnjcoo}} {{tag|cnjadv}} {{tag|cnjsub}} || -     || - || - || Create ar verb script based on mt, add ar open-class words according to frequency
 |-
-| 3       || <s>04/06&mdash;10/06</s>  ||   59%           || 75.3%, 74.4% ||  {{tag|adv}} || 200 words    || 8.70 % || Continue adding open class words to ar (adding to bidix/mt as you go)
+| 3       || <s>04/06&mdash;10/06</s>  ||   59%           || 75.3%, 74.4% ||  {{tag|adv}} || 200 words    || 8.70 % || 20.94 % || Continue adding open class words to ar (adding to bidix/mt as you go)
 |-
-| 4       || <s>11/06&mdash;17/06</s>  ||   63%           || 76.0%, 75.1% ||  {{tag|prn}} {{tag|det}} || -   || - || Monodix completion; closed classes (apart from verb-attached pronouns) should be testvoc clean.
+| 4       || <s>11/06&mdash;17/06</s>  ||   63%           || 76.0%, 75.1% ||  {{tag|prn}} {{tag|det}} || -   || - || - || Monodix completion; closed classes (apart from verb-attached pronouns) should be testvoc clean.
 |-
-| 5       || <s>18/06&mdash;24/06</s>  ||   68%           || 79.0%, 78.3% ||  {{tag|adj}} || -   || - || Bidix expansion; bidix adjectives should be testvoc clean.
+| 5       || <s>18/06&mdash;24/06</s>  ||   68%           || 79.0%, 78.3% ||  {{tag|adj}} || -   || - || - || Bidix expansion; bidix adjectives should be testvoc clean.
 |-
-| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11% || '''Midterm evaluation'''. Bidix expansion, multiwords.
+| 6       || <s>25/06&mdash;01/07</s>  ||   70%           || 79.1%, 78.4% ||  {{tag|n}} {{tag|np}} || 500 words   || 23.11% || 33.53 % || '''Midterm evaluation'''. Bidix expansion, multiwords.
 |-
-| 7       || <s>02/07&mdash;08/07</s>  ||   73%           || 79.4%, 78.6% ||  - || -   || - || Transfer
+| 7       || <s>02/07&mdash;08/07</s>  ||   73%           || 79.4%, 78.6% ||  - || -   || - || - || Transfer
 |-
-| 8       || <s>09/07&mdash;15/07</s>  ||   75%           || 79.5%, 78.8% ||  - || -   || - || Transfer, corpus testvoc
+| 8       || <s>09/07&mdash;15/07</s>  ||   75%           || 79.5%, 78.8% ||  - || -   || - || - || Transfer, corpus testvoc
 |-
-| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28% || Disambiguation, retrain tagger
+| 9       || <s>16/07&mdash;22/07</s>  ||   77%           || 80.5%, 80.0% ||  - || 200 words   || 17.28% || 35.89 % || Disambiguation, retrain tagger
 |-
-| 10      || <s>23/07&mdash;29/07</s>  ||   80%           || 80.9%, 80.2% ||  {{tag|v}} || -  || - || Testvoc
+| 10      || <s>23/07&mdash;29/07</s>  ||   80%           || 80.9%, 80.2% ||  {{tag|v}} || -  || - || - || Testvoc
 |-
-| 11      || 30/07&mdash;05/08  ||   80%           || - || - || -  || - || Testvoc
+| 11      || 30/07&mdash;05/08  ||   80%           || - || - || -  || - || - || Testvoc
 |-
-| 12      || 06/08&mdash;12/08  ||   80%           || - || ''all categories clean'' || 500 words  || ? || '''Final evaluation'''. Tidying up, releasing
+| 12      || 06/08&mdash;12/08  ||   80%           || - || ''all categories clean'' || 500 words  || ? || - || '''Final evaluation'''. Tidying up, releasing
 |-
 |}