Bengali and English/Updating Bilingual Dictionary - Revision history

Darthxaher at 13:05, 25 August 2009

2009-08-25T13:05:02Z

Darthxaher at 13:03, 25 August 2009

2009-08-25T13:03:46Z

Darthxaher at 12:47, 25 August 2009

2009-08-25T12:47:13Z

Darthxaher at 12:32, 25 August 2009

2009-08-25T12:32:23Z

Darthxaher at 12:27, 25 August 2009

2009-08-25T12:27:09Z

Darthxaher at 12:25, 25 August 2009

2009-08-25T12:25:57Z

Darthxaher at 12:16, 25 August 2009

2009-08-25T12:16:32Z

Darthxaher at 12:11, 25 August 2009

2009-08-25T12:11:31Z

Darthxaher at 12:01, 25 August 2009

2009-08-25T12:01:46Z

Darthxaher at 05:14, 25 August 2009

2009-08-25T05:14:38Z

@@ Line 49: / Line 49: @@
 cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f1 | perl -pe 's/ /<b\/>/g' > /tmp/bar4 && cat /tmp/bar3 | cut -f2 |
 perl -pe 's/ /<b\/>/g' > /tmp/bar5
-paste /tmp/bar4 /tmp/bar5 | perl -pe 's/<(\w+)>/<s n="$1"\/>/g' | awk -F'\t' '{ print "    <e><p><l>"$1"</l><r>"$2"<s n=\"n\"/></r></p></e>" }'
+paste /tmp/bar4 /tmp/bar5 | perl -pe 's/<(\w+)>/<s n="$1"\/>/g' | awk -F'\t' '{ print "    <e><p><l>"$1"</l><r>"$2"</r></p></e>" }'
 </pre>

@@ Line 47: / Line 47: @@
 We can now try merging English and Bengali tagged parts to bdix entries, we'll also need change the tags to bdix format.
 <pre>
-cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f1 > /tmp/bar4 && cat /tmp/bar3 | cut -f2 > /tmp/bar5
+cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f1 | perl -pe 's/ /<b\/>/g' > /tmp/bar4 && cat /tmp/bar3 | cut -f2 |
+perl -pe 's/ /<b\/>/g' > /tmp/bar5
 paste /tmp/bar4 /tmp/bar5 | perl -pe 's/<(\w+)>/<s n="$1"\/>/g' | awk -F'\t' '{ print "    <e><p><l>"$1"</l><r>"$2"<s n=\"n\"/></r></p></e>" }'
 </pre>

@@ Line 44: / Line 44: @@
 Now we open <code>/tmp/bar3</code> file, check the entries, specially the ones marked as only <code><adj></code>, we need to mark them also with <code><sint></code> if they are synthetic.
+We can now try merging English and Bengali tagged parts to bdix entries, we'll also need change the tags to bdix format.
+<pre>
+cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f1 > /tmp/bar4 && cat /tmp/bar3 | cut -f2 > /tmp/bar5
+paste /tmp/bar4 /tmp/bar5 | perl -pe 's/<(\w+)>/<s n="$1"\/>/g' | awk -F'\t' '{ print "    <e><p><l>"$1"</l><r>"$2"<s n=\"n\"/></r></p></e>" }'
+</pre>
+The output will look something like this.
+<pre>
+    <e><p><l>চিহ্নিত<s n="adj"/><s n="mf"/></l><r>marked<s n="adj"/></r></p></e>
+    <e><p><l>ভিন্ন<s n="adj"/><s n="mf"/></l><r>different<s n="adj"/></r></p></e>
+    <e><p><l>শিক্ষিত<s n="adj"/><s n="mf"/></l><r>educated<s n="adj"/></r></p></e>
+    <e><p><l>অন্তর্ভুক্ত<s n="adj"/><s n="mf"/></l><r>included<s n="adj"/></r></p></e>
+</pre>
+We can now safely add these entries to the desired section of bdix.

@@ Line 40: / Line 40: @@
 perl -pe 's/\$//g' > /tmp/bar2 && paste /tmp/bar1 /tmp/bar2 | awk '{ if($2 ~ /<adj>/) { print $0 } else { print $1"\t"$1"<adj>" } }' > /tmp/bar3
 </pre>
+Surely, there will be entries which are not in the en-es monodix, we know for sure that they are adjectives, so we try tagging them with <code><adj></code>.
+Now we open <code>/tmp/bar3</code> file, check the entries, specially the ones marked as only <code><adj></code>, we need to mark them also with <code><sint></code> if they are synthetic.

@@ Line 36: / Line 36: @@
 <pre>
-cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f2 | tee /tmp/bar1 |  lt-proc ../../trunk/apertium-en-es-0.6/en-es.automorf.bin |  perl -pe 's/(\^)(.*?)(\/)//' | perl -pe 's/<comp>|<sup>//' | perl -pe 's/\/?\w*?(<vblex>|<n>|<adv>|<pr>|<prn>|<np>|<predet>)(<\w+>)*(\/|\$)?//g' | perl -pe 's/\$//g' > /tmp/bar2 && paste /tmp/bar1 /tmp/bar2 | awk '{ if($2 ~ /<adj>/) { print $0 } else { print $1"\t"$1"<adj>" } }' > /tmp/bar3
+cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f2 | tee /tmp/bar1 |  lt-proc ../../trunk/apertium-en-es-0.6/en-es.automorf.bin |  \
+perl -pe 's/(\^)(.*?)(\/)//' | perl -pe 's/<comp>|<sup>//' | perl -pe 's/\/?\w*?(<vblex>|<n>|<adv>|<pr>|<prn>|<np>|<predet>)(<\w+>)*(\/|\$)?//g' | \
+perl -pe 's/\$//g' > /tmp/bar2 && paste /tmp/bar1 /tmp/bar2 | awk '{ if($2 ~ /<adj>/) { print $0 } else { print $1"\t"$1"<adj>" } }' > /tmp/bar3
 </pre>

@@ Line 31: / Line 31: @@
 </pre>
-We can add these entries with the help of a an external en-bn dix, right now we are adding them manually. If you look closely, the entries are tab separated and there are few special characters. <code>#</code> is for any unknown entry, <code>!</code> for a different pos (here, the entry 'শাসক' was mistakenly tagged adjective in the monodix) and <code>?</code> is for any other confusion. We are going to discard any entry with any of these symbols in later processing.
+We can add these entries with the help of a an external en-bn dix, right now we are adding them manually. If you look closely, the entries are tab separated and there are few special characters. <code>#</code> is for any unknown entry, <code>!</code> for a different pos (here, the entry 'শাসক' was mistakenly tagged adjective in the monodix) and <code>?</code> is for any other confusion. We are going to discard any entry with any of these symbols in any subsequent processing.
+After adding all the entries, we can try pos-tagging the English words. We're now going to use en-es dix for this purpose, since it has more entries in the eng monodix than ours.
+<pre>
+cat dev/bdix/adjective.list | egrep -v '\#|\?|\!' | cut -f2 | tee /tmp/bar1 |  lt-proc ../../trunk/apertium-en-es-0.6/en-es.automorf.bin |  perl -pe 's/(\^)(.*?)(\/)//' | perl -pe 's/<comp>|<sup>//' | perl -pe 's/\/?\w*?(<vblex>|<n>|<adv>|<pr>|<prn>|<np>|<predet>)(<\w+>)*(\/|\$)?//g' | perl -pe 's/\$//g' > /tmp/bar2 && paste /tmp/bar1 /tmp/bar2 | awk '{ if($2 ~ /<adj>/) { print $0 } else { print $1"\t"$1"<adj>" } }' > /tmp/bar3
+</pre>

@@ Line 11: / Line 11: @@
 we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>|&lt;sup&gt;//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's <code>'uniq'</code>, which is not fully Unicode compliant.
-Assume that we have this output saved in <code>dev/bdix/adjective.list</code> file. Let's see how the file looks like in the first glance.
+Assume that we have this output saved in <code>dev/bdix/adjective.list</code> file. Let's see how the file looks like at the first glance.
 <pre>
 চিহ্নিত<adj><mf>
@@ Line 30: / Line 30: @@
 অন্তর্ভুক্ত<adj><mf>    included
 </pre>
+We can add these entries with the help of a an external en-bn dix, right now we are adding them manually. If you look closely, the entries are tab separated and there are few special characters. <code>#</code> is for any unknown entry, <code>!</code> for a different pos (here, the entry 'শাসক' was mistakenly tagged adjective in the monodix) and <code>?</code> is for any other confusion. We are going to discard any entry with any of these symbols in later processing.

@@ Line 9: / Line 9: @@
 </pre>
-we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>|&lt;sup&gt;//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's <code>'uniq'</code>, which is not fully unicode compliant.
+we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>|&lt;sup&gt;//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's <code>'uniq'</code>, which is not fully Unicode compliant.
+Assume that we have this output saved in <code>dev/bdix/adjective.list</code> file. Let's see how the file looks like in the first glance.
+<pre>
+চিহ্নিত<adj><mf>
+মঞ্চস্থ<adj><mf>
+ভিন্ন<adj><mf>
+শিক্ষিত<adj><mf>
+শাসক<adj><mf>
+অন্তর্ভুক্ত<adj><mf>
+</pre>
+Now we are going to add corresponding English entries to this file. So after adding entries the file looks like this
+<pre>
+চিহ্নিত<adj><mf>    marked
+মঞ্চস্থ<adj><mf>    #
+ভিন্ন<adj><mf>    different
+শিক্ষিত<adj><mf>    educated
+শাসক<adj><mf>    ruler    !
+অন্তর্ভুক্ত<adj><mf>    included
+</pre>

← Older revision		Revision as of 12:01, 25 August 2009
Line 9:		Line 9:
	</pre>		</pre>

	we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>\|<sup>//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's 'uniq', which is not fully unicode compliant.		we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>\|<sup>//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's <code>'uniq'</code>, which is not fully unicode compliant.

@@ Line 1: / Line 1: @@
-Assuming that we are in the apertium-bn-en folder
+We are going to try adding more adjective entries in the bn-en bdix. Assuming that we are in the apertium-bn-en folder (download it from the svn), try this,
 <pre>
@@ Line 8: / Line 8: @@
 perl -pe 's/<comp>|<sup>//g' | python dev/uniq.py
 </pre>
+we used <code>grep '<adj>'</code> to filter out the adjectives, and <code>perl -pe 's/<comp>|<sup>//g'</code> to remove the tags inflection tags from every adjective entry. Then we used <code>uniq.py</code> to filter the uniq entries instead of shell's 'uniq', which is not fully unicode compliant.