Ilnar.salimzyan at 00:10, 18 March 2012

2012-03-18T00:10:57Z

Ilnar.salimzyan: Created page with '== Intention == This little script aims to facilitate adding new words to dictionaries of a language pair which uses HFST. == Usage == It is intended to be used the following w…'

2012-03-17T22:04:30Z

Created page with '== Intention == This little script aims to facilitate adding new words to dictionaries of a language pair which uses HFST. == Usage == It is intended to be used the following w…'

New page

== Intention ==

This little script aims to facilitate adding new words to dictionaries of a language pair which uses HFST.

== Usage ==
It is intended to be used the following way.

Suppose that morphological transducer for Tatar in Tatar-Bashkir pair doesn't recognise the word 'укытучы', which means "teacher". We have to add this word to:

# <code>apertium-tt-ba.tt.lexc</code>,
# translate it in <code>apertium-tt-ba.tt-ba.dix</code> and
# add the translation to <code>apertium-tt-ba.ba.lexc</code>. For such close related languages like Tatar and Bashkir most likely all the categories and the names of continuation classes will remain the same.

In a directory containing <code>lexikograf.sh</code> there should be a file called <code>add-them.txt</code>, to which you add words exactly in the same form as you would add them to lexc file of a given language, in our case

<code>
укытучы NLEX ;
</code><ref>Actually <code>укытучы:укытучы Ninfl ;</code> is a better way of writing this and statement above that you should add stems to ''add-them.txt'' exactly in the same form as you would add them to the actual .lexc file is not absolutely true. But such writing obviously saves time. Don't worry, lexikograf.sh will generate entries in the preferred format</ref>

After it, if you run lexikograf.sh it will generate three text files: <code>to-tatar-lexc.txt</code>, <code>to-bidix.txt</code> and <code>to-bashkir-lexc.txt</code>. In our case, each file will contain only one line:

<code>
!укытучы NLEX ; ! ""
</code>

<code>
<nowiki><e><l>укытучы<s n="n"/></l><r><s n="n"/></r></e></nowiki>
</code>

<code>
! NLEX ; ! "укытучы"
</code>
respectively.

After it, one obviously has to do the following:
* add Bashkir translations in <code>to-bashkir-lexc.txt</code> (and additional tags if they aren't matching entirely, e.g. different gender, difference in transitivity of verbs etc. Hard to imagine in Tatar-Bashkir pair actually):
** <code><nowiki><e><l>укытучы<s n="n"/></l><r>уҡытыусы<s n="n"/></r></e></nowiki></code>
* add this translation to <code>to-bashkir-lexc.txt</code>. Again, you can write the word only once, the next script, <code>duplicate.sh</code> will take care of it and make from entries <code>!укытучы NLEX ; ! ""</code> and <code>!уҡытыусы NLEX ; ! "укытучы"</code> entries <code>!укытучы:укытучы NLEX ; ! ""</code> and <code>!уҡытыусы:уҡытыусы NLEX ; ! "укытучы"</code> respectively. This lines remain commented out so that you cannot add them to the actual .lexc files without proofreading them.

== Code ==
Here is the script itself. If you like the idea of it and think that it can indeed be usefull, feel free to modify it, as it was written 1. by an absolute newbie and 2. won't work with any other language pair without modifying it. Define some variables etc. Be bold :)

<pre>
#!/bin/bash

# This little tools aims to faciltate adding new stems to dictionaries of a language pair using HFST
# by reducing amount of typing work
# see http://wiki.apertium.org/wiki/Lexikograf.sh for more details

# for every line of add-them.txt
#do

## match the first word (=initial letters till the first space) of the line and store it in $one
### the first space can be escaped by a %. In such case skip it and match all letters till the second space

## match rest of the line to $rest

# print '$LINE ! ""' to to-tatar-lexc.txt

# print '<nowiki><e><l>$one<s n="n"/></l><r><s n="n"/></r></e></nowiki>" to to-bidix.txt

# print '$rest ! "$one"' to to-bashkir.lexc

# duplicate words in lexc entries and comment them out
</pre>

==Notes==
<references/>

@@ Line 6: / Line 6: @@
 It is intended to be used the following way.
-Suppose that morphological transducer for Tatar in Tatar-Bashkir pair doesn't recognise the word 'укытучы', which means "teacher". We have to add this word to:
+Suppose that morphological transducer for Tatar in Tatar-Bashkir pair doesn't recognize the word 'укытучы', which means "teacher". We have to add this word to:
 # <code>apertium-tt-ba.tt.lexc</code>,
+# translate it in <code>apertium-tt-ba.tt-ba.dix</code><ref>We are adding a word which is not recognized by the morphological analyzer of Tatar, and therefore not present in tt.lexc. But this does not necessarily mean that this word and it's translation are also missing in tt-ba.dix and ba.lexc. So it will be a good idea to check it first and only then generate entries</ref> and
-# translate it in <code>apertium-tt-ba.tt-ba.dix</code> and
 # add the translation to <code>apertium-tt-ba.ba.lexc</code>. For such close related languages like Tatar and Bashkir most likely all the categories and the names of continuation classes will remain the same.
@@ Line 26: / Line 26: @@
 <code>
 <nowiki><e><p><l>укытучы<s n="n"/></l><r><s n="n"/></r></p></e></nowiki>
+</code><ref>The tags used in tt-ba.dix should be matched against Root Lexicon names, so that script knows, which symbol it has to put in bidix inside of <nowiki><s n="?"></nowiki>. A simpler solution would be to leave this tags empty, but this is definitively not good</ref>
-</code>
 <code>
@@ Line 39: / Line 39: @@
 == Code ==
-Here is the script itself. If you like the idea of it and think that it can indeed be usefull, feel free to modify it, as it was written 1. by an absolute newbie and 2. won't work with any other language pair without modifying it. Define some variables etc. Be bold :)
+Here is the script itself. If you like the idea of it and think that it indeed could be useful, feel free to modify it. Define some variables, so it can be used with any language pair etc. Be bold :)
 <pre>
 #!/bin/bash
-# This little tools aims to faciltate adding new stems to dictionaries of a language pair using HFST
+# This little tools aims to facilitate adding new stems to dictionaries of a language pair using HFST
 # by reducing amount of typing work
 # see http://wiki.apertium.org/wiki/Lexikograf.sh for more details

Lexikograf.sh - Revision history

Ilnar.salimzyan at 00:10, 18 March 2012

Ilnar.salimzyan: Created page with '== Intention == This little script aims to facilitate adding new words to dictionaries of a language pair which uses HFST. == Usage == It is intended to be used the following w…'