Talk:Turkic-Turkic translator - Revision history

Francis Tyers at 18:54, 12 May 2012

2012-05-12T18:54:36Z

Francis Tyers at 12:19, 23 April 2012

2012-04-23T12:19:41Z

Francis Tyers at 12:54, 22 April 2012

2012-04-22T12:54:21Z

Francis Tyers: Created page with ' ==Lexicon trimming==

  also, some 'corner cases' for the lexicon scraper  when you have stems in a continuation lexicon, e.g. demonstratives in kazakh, p…'

2012-04-22T12:52:19Z

Created page with ' ==Lexicon trimming== <pre> <spectre> also, some 'corner cases' for the lexicon scraper <spectre> when you have stems in a continuation lexicon, e.g. demonstratives in kazakh, p…'

New page

==Lexicon trimming==

<pre>
<spectre> also, some 'corner cases' for the lexicon scraper
<spectre> when you have stems in a continuation lexicon, e.g. demonstratives in kazakh, personal pronouns in other places
<spectre> they will all get included disregarding the bilingual dictionary
<spectre> another example: when one language has a word that another doesn't have (e.g. a case that turns into a postposition,
but the postposition doesn't have an equivalent in the other language, e.g. is inserted by transfer
<spectre> (same goes for "particles")
<spectre> for verbs (this one can be fixed by having a correspondence between tags/continuation lexica)
e.g. when you have a verb which is both tv/iv but only one in the bidix
</pre>

@@ Line 21: / Line 21: @@
 * Treat clitics like 'mi', 'i' etc. separately, and not as attached. In Turkish this would reduce the size of those categories which can take these clitics by _at least_ 6 times.
 * Because of how the lexicons are laid out. We could try and do some kind of continuation-based testvoc.
+Idea:
+* Read in the lexc file, and from the stems, reading up, make a list of the combinations of continuation lexicons, e.g. <code>V-TV V-FIN-COMMON V-NONFIN ...</code>
+* Make a hash relating each combination of continuation lexicons to a list of stems
+* Do this for both lexc files
+* Match up the combinations via the bilingual dictionary. e.g. <code>"foo" N -- "bar" N; "baz" N-NOPOS - "barm" N</code>
+* Then for each combination of lists of continuation lexicons, make lists of the different combinations with the bilingual dictionary.
+* Select <math>n</math> at random from each of the pairs and expand them.

@@ Line 13: / Line 13: @@
           for foo<adj><subst>:bar<n> (or foo<adj><subst>:baz<n>) then there will be errors
 </pre>
+==Testvoc==
+We probably need to work out a way to run the testvoc in a reasonable amount of time.  Here are some suggestions:
+* Create sub-lexicons, which just run one category through the testvoquing process.
+* Treat clitics like 'mi', 'i' etc. separately, and not as attached. In Turkish this would reduce the size of those categories which can take these clitics by _at least_ 6 times.
+* Because of how the lexicons are laid out. We could try and do some kind of continuation-based testvoc.

@@ Line 1: / Line 1: @@
 ==Lexicon trimming==
@@ Line 11: / Line 10: @@
 <spectre> for verbs (this one can be fixed by having a correspondence between tags/continuation lexica)
           e.g. when you have a verb which is both tv/iv but only one in the bidix
+<spectre> another example: when you have an entry like foo<adj>:bar<n><attr> in the bilingual dictionary, but no entry
+          for foo<adj><subst>:bar<n> (or foo<adj><subst>:baz<n>) then there will be errors
 </pre>