User:Ilnar.salimzyan/GSoC2014 - Revision history

Ilnar.salimzyan: /* sme-nob paper, eus-eng paper, eng-kaz paper, “Good Applications for Crummy Machine Translation" */

2014-06-11T13:08:44Z

sme-nob paper, eus-eng paper, eng-kaz paper, “Good Applications for Crummy Machine Translation"

Ilnar.salimzyan: /* sme-nob paper, eus-eng paper, eng-kaz paper */

2014-06-11T13:08:25Z

sme-nob paper, eus-eng paper, eng-kaz paper

Ilnar.salimzyan at 09:50, 3 June 2014

2014-06-03T09:50:03Z

Ilnar.salimzyan: /* Testvoc */

2014-06-03T09:43:39Z

Testvoc

Ilnar.salimzyan: /* Community-bonding period */

2014-06-03T09:39:36Z

Community-bonding period

Ilnar.salimzyan: /* Testvoc */

2014-06-03T09:29:10Z

Testvoc

Ilnar.salimzyan: /* Chunking, Chunking: A full example, N-Stage transfer */

2014-06-03T09:20:42Z

Chunking, Chunking: A full example, N-Stage transfer

Unhammer: /* Chunking, Chunking: A full example, N-Stage transfer */

2014-05-26T07:17:29Z

Chunking, Chunking: A full example, N-Stage transfer

Ilnar.salimzyan: /* Literature review */

2014-05-19T22:16:49Z

Literature review

Ilnar.salimzyan: /* Community-bonding period */

2014-05-18T11:00:42Z

Community-bonding period

@@ Line 32: / Line 32: @@
 *: if possible, yes :) --[[User:Unhammer|unhammer]] ([[User talk:Unhammer|talk]]) 09:17, 26 May 2014 (CEST)
-==== <s>sme-nob paper</s>, eus-eng paper, eng-kaz paper, “Good Applications for Crummy Machine Translation" ====
+==== <s>sme-nob paper</s>, eus-eng paper, eng-kaz paper, <s>“Good Applications for Crummy Machine Translation"</s> ====
 * Use macros, they make transfer files shorter and therefore more comprehensible.
 * It's possible to use twol to forbid some analyses (hopefully we won't have to use that, but good to know that it's possible).

@@ Line 32: / Line 32: @@
 *: if possible, yes :) --[[User:Unhammer|unhammer]] ([[User talk:Unhammer|talk]]) 09:17, 26 May 2014 (CEST)
-==== <s>sme-nob paper</s>, eus-eng paper, eng-kaz paper ====
+==== <s>sme-nob paper</s>, eus-eng paper, eng-kaz paper, “Good Applications for Crummy Machine Translation" ====
 * Use macros, they make transfer files shorter and therefore more comprehensible.
 * It's possible to use twol to forbid some analyses (hopefully we won't have to use that, but good to know that it's possible).

@@ Line 6: / Line 6: @@
 This page is used to organize thoughts and document the development process. If you are only interested in the workplan and stats, refer to the 'Workplan' and 'Current state' sections of the [[Tatar and Russian]] page.
-== Post-application period ==
 <pre>
-* work on the 'James and Mary' translation
+* <s>work on the 'James and Mary' translation
     ** get rid of the debugging symbols
-    ** get the baseline WER
+    ** get the baseline WER</s>
 * get permission to use one of the modern government-funded Tatar-Russian
   dictionaries under a free license and digitize it or fall back to one of
@@ Line 37: / Line 35: @@
 * Use macros, they make transfer files shorter and therefore more comprehensible.
 * It's possible to use twol to forbid some analyses (hopefully we won't have to use that, but good to know that it's possible).
-'''Other thoughts:'''
-* Acceptance tests for an Aperitum MT system are: regression tests on the wiki, corpus test (WER and number of [*@#] errors) and testvoc. Unit testing an Apertium MT system is testing its modules (modes). Figure out how to unit test each module.
-** One should be able to run his tests without the internet connection. Keeping a copy of the 'regression tests' html page in the /dev solves this problem, but it doesn't allow us to add new tests while not having internet access. One way to deal with that is to have a local copy of regression tests in the wiki format, so that if you add new test while flying over the atlantic, you can copy paste them to the wiki page of the pair later.
-* Try to avoid variables (i.e. use attributes where possible, not variables).
-** E.g. in a numeral_noun agreement rule, you want numeral's gender and case to be the same as noun's gender and case. You can of course assign nouns attribute values to variables and use these variables for the numeral. But the problem is that numerals can have different number of tags (e.g. digits only receive the <num> tag).
-*** <s>"Adapt" macros as seen in r53019 of tat-rus can help with that</s> (they give numerals some default tags (different number of them, based on what kind of numeral it is): <pre>бер<num>:один<num><m><an><sg><nom>;  биш<num>:пять<num><mfn><pl><nom>;  1<num>:1<num></pre> so that you can "let numeral's atribute be like noun's attribute" later in a rule).
-*** Use [[Placeholder attributes]] (CD, AD etc.) instead: <pre>бер<num>:один<num><GD><AD><ND><CD>;  биш<num>:пять<num><mfn><pl><CD>;  1<num>:1<num></pre> They accomplish the same thing (one numeral_noun rule will work in all three cases), but in a much simpler way.
-== Community-bonding period ==
 <pre>
@@ Line 81: / Line 67: @@
 See apertium-tat-rus/tests/t1x.test ('./qa.sh t1x' typed in apertium-tat-rus/ will run it).
+== Other thoughts ==
+* Acceptance tests for an Aperitum MT system are: regression tests on the wiki, corpus test (WER and number of [*@#] errors) and testvoc. Unit testing an Apertium MT system is testing its modules (modes). Figure out how to unit test each module.
+** One should be able to run his tests without the internet connection. Keeping a copy of the 'regression tests' html page in the /dev solves this problem, but it doesn't allow us to add new tests while not having internet access. One way to deal with that is to have a local copy of regression tests in the wiki format, so that if you add new test while flying over the atlantic, you can copy paste them to the wiki page of the pair later.
+* Try to avoid variables (i.e. use attributes where possible, not variables).
+** E.g. in a numeral_noun agreement rule, you want numeral's gender and case to be the same as noun's gender and case. You can of course assign nouns attribute values to variables and use these variables for the numeral. But the problem is that numerals can have different number of tags (e.g. digits only receive the <num> tag).
+*** <s>"Adapt" macros as seen in r53019 of tat-rus can help with that</s> (they give numerals some default tags (different number of them, based on what kind of numeral it is): <pre>бер<num>:один<num><m><an><sg><nom>;  биш<num>:пять<num><mfn><pl><nom>;  1<num>:1<num></pre> so that you can "let numeral's atribute be like noun's attribute" later in a rule).
+*** Use [[Placeholder attributes]] (CD, AD etc.) instead: <pre>бер<num>:один<num><GD><AD><ND><CD>;  биш<num>:пять<num><mfn><pl><CD>;  1<num>:1<num></pre> They accomplish the same thing (one numeral_noun rule will work in all three cases), but in a much simpler way.
 == References ==

@@ Line 68: / Line 68: @@
 Ok, the usual testvoc (see apertium-tat-rus/testvoc/standard) works and so far doesn't take too much time to run.
-We've also set up prefixing system in apertium-tat/tests/morphotactics which, for one word per pardef, provides a text file with the full paradigm of that word. The apertium-tat-rus/testvoc/lite, which is supposed to take that text files from apertium-tat, extract LU's, run them through inconsistency.sh and generate testvoc-summary file, <s>where each line represents stats about each text file,</s> works as well and runs pretty fast. It generates stats for each category (i.e. nouns, adjectives etc.), not each lexicon name in apertium-tat/tests/morphotactics (i.e. N1, N-COMPOUND-PX etc.)
+We've also set up prefixing system in apertium-tat/tests/morphotactics which, for one word per pardef, provides a text file with the full paradigm of that word. The apertium-tat-rus/testvoc/lite, which is supposed to take that text files from apertium-tat, extract LU's, run them through inconsistency.sh and generate testvoc-summary file, <s>where each line represents stats about each text file,</s> works as well and runs pretty fast. It uses standard testvoc's <code>inconsistency.sh</code> and <code>inconsistency-summary.sh</code> scripts and thus generates stats for each category (i.e. nouns, adjectives etc.), not each lexicon name in apertium-tat/tests/morphotactics (i.e. N1, N-COMPOUND-PX etc.)
 Corpus testvoc script is in apertium-tat-rus/testvoc/corpus.

@@ Line 60: / Line 60: @@
 # multiword pending tests on the wiki which kind of cover the core of the desired functionality
   (at least 52 "sentence models" listed in the "Tatar Syntax" book)
-# "workplan" and "current state" tables on [[Tatar and Russian]] page which will track progress
+# <s>"workplan" and "current state" tables on [[Tatar and Russian]] page which will track progress
-  on things I've promised to do in the proposal (on week end, 17-18 May)
+  on things I've promised to do in the proposal (on week end, 17-18 May)</s>
 </pre>

@@ Line 66: / Line 66: @@
 === Testvoc ===
-Ok, the usual testvoc (see apertium-tat-rus/testvoc/standard) works and so far doesn't take too much time to run. We've also set up prefixing system in apertium-tat/tests/morphotactics which, for one word per pardef, provides a text file with the full paradigm of that word.
+Ok, the usual testvoc (see apertium-tat-rus/testvoc/standard) works and so far doesn't take too much time to run.
-The apertium-tat-rus/testvoc/lite, which is supposed to take that text files from apertium-tat, extract LU's, run them through inconsistency.sh and generate testvoc-summary file where each line represents stats about each text file doesn't work yet. For the time being, I can can do that manually, simply grep for "[@#]" errors and automate it along the way.
+We've also set up prefixing system in apertium-tat/tests/morphotactics which, for one word per pardef, provides a text file with the full paradigm of that word. The apertium-tat-rus/testvoc/lite, which is supposed to take that text files from apertium-tat, extract LU's, run them through inconsistency.sh and generate testvoc-summary file, <s>where each line represents stats about each text file,</s> works as well and runs pretty fast. It generates stats for each category (i.e. nouns, adjectives etc.), not each lexicon name in apertium-tat/tests/morphotactics (i.e. N1, N-COMPOUND-PX etc.)
-If time permits, would be good to set it up sooner rather than later (and also the corpus testvoc in the same apertium-tat-rus/testvoc directory).
+Corpus testvoc script is in apertium-tat-rus/testvoc/corpus.
 === A way to test the Russian generator ===

@@ Line 30: / Line 30: @@
 (Ideas to try out and notes)
-==== <s>[[Chunking]]</s>, <s>[[Chunking: A full example]]</s>, [[N-Stage transfer]] ====
+==== <s>[[Chunking]]</s>, <s>[[Chunking: A full example]]</s>, <s>[[N-Stage transfer]]</s> ====
 * Probably I should stick to the SN and SV convention?
 *: if possible, yes :) --[[User:Unhammer|unhammer]] ([[User talk:Unhammer|talk]]) 09:17, 26 May 2014 (CEST)

@@ Line 51: / Line 51: @@
 <pre>
 '''Deliverables 0:'''
-# testvoc script(s) which doesn't take forever to run (consider footnote #5 in the proposal)
+# <s>testvoc script(s) which doesn't take forever to run (consider footnote #5 in the proposal)</s>
-# a way to test the apertium-rus generator
+# <s>a way to test the apertium-rus transducer</s>
 # a digital dictionary under a free license or ocr'd public domain dictionary
 # parallel corpus in /corpus (=development corpus) is expanded with texts which represent domains
   the system could potentially be applied to (500 sentences?)
-# t1x.test, t2x.test, t3x.test and transfer.test which will run
+# <s>t1x.test, t2x.test, t3x.test and transfer.test which will run all three</s>
-  all three
 # multiword pending tests on the wiki which kind of cover the core of the desired functionality
   (at least 52 "sentence models" listed in the "Tatar Syntax" book)