User:Ragib06/GSoC2011 Challenge Answer - Revision history

Ragib06: /* Answer : More (100 new nouns) */

2011-04-13T02:50:25Z

Answer : More (100 new nouns)

Ragib06: /* Answer : More (100 new nouns) */

2011-04-13T02:46:08Z

Answer : More (100 new nouns)

Ragib06: /* Answer : More (100 new nouns) */

2011-04-11T21:07:11Z

Answer : More (100 new nouns)

Ragib06 at 20:45, 11 April 2011

2011-04-11T20:45:49Z

115.127.15.171: /* Answer */

2011-04-09T07:54:41Z

Answer

115.127.15.171: /* Answer */

2011-04-07T06:35:46Z

Answer

115.127.15.171: /* Answer */

2011-04-07T04:39:02Z

Answer

Ragib06: /* Answer */

2011-04-06T19:27:22Z

Answer

Ragib06: /* Answer */

2011-04-06T19:24:52Z

Answer

Ragib06: Created page with '== Challenge 1: Updating Bengali-English Bilingual Dictionary == Difficulty: Easy Title: Update the Bengali English Bilingual Dictionary with 100 new entries. Instructions on ho…'

2011-04-06T17:58:37Z

Created page with '== Challenge 1: Updating Bengali-English Bilingual Dictionary == Difficulty: Easy Title: Update the Bengali English Bilingual Dictionary with 100 new entries. Instructions on ho…'

New page

== Challenge 1: Updating Bengali-English Bilingual Dictionary ==
Difficulty: Easy

Title: Update the Bengali English Bilingual Dictionary with 100 new entries. Instructions on how to add more entries are described in the article [[Bengali_and_English/Updating Bilingual Dictionary|Updating Bilingual Dictionary]]

Deliverable: A patch file

== Answer ==
Assuming that we are in the '''apertium-bn-en''' folder, we can run this to extract the adjectives from '''apertium-bn-en.bn.dix''' :
<pre>
lt-expand apertium-bn-en.bn.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | perl -pe 's/<comp>|//g' | python dev/uniq.py
</pre>
There are 866 adjectives expanded. Now, to get the adjective list from the current system we can run this (according to the article mentioned above):
<pre>
lt-expand apertium-bn-en.bn.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | tee /tmp/foo1 | sed 's/^/^/g' | sed 's/$/$/g' | \
sed 's/$/^.<sent>$/g' | apertium-pretransfer | apertium-transfer apertium-bn-en.bn-en.t1x bn-en.t1x.bin bn-en.autobil.bin | \
apertium-interchunk apertium-bn-en.bn-en.t2x bn-en.t2x.bin | apertium-postchunk apertium-bn-en.bn-en.t3x bn-en.t3x.bin | tee /tmp/foo2 | \
lt-proc -g bn-en.autogen.bin > /tmp/foo3 && paste /tmp/foo1 /tmp/foo2 /tmp/foo3 | egrep -v '\+' | egrep -v '@' | cut -f1 | \
perl -pe 's/<comp>|//g' | python dev/uniq.py
</pre>
And we get 87 adjectives. Now, saving the adjective lists, we can run a simple script to find around 100 adjetives from those 866-87 = 779 non-listed entries. Say, we save these new adjectives into a file "newadj.txt". Now, we need to put the lemma mappings manually.
A sample may be like this:
<pre>
অজানা<adj><mf> unknown
তরল<adj><mf> liquid
মানসিক<adj><mf> mental
অসহায়<adj><mf> helpless
</pre>
Now, for pos tagging we can again use the code from the mentioned article:
<pre>
cat newadj.txt | egrep -v '\#|\?|\!' | cut -f2 | tee /tmp/bar1 | lt-proc en-bn.automorf.bin | \
perl -pe 's/(\^)(.*?)(\/)//' | perl -pe 's/<comp>|//' | perl -pe 's/\/?\w*?(<vblex>|<n>|<adv>|<pr>|<prn>|<np>|<predet>)(<\w+>)*(\/|\$)?//g' | \
perl -pe 's/\$//g' > /tmp/bar2 && paste /tmp/bar1 /tmp/bar2 | awk '{ if($2 ~ /<adj>/) { print $0 } else { print $1"\t"$1"<adj>" } }' > /tmp/bar3
</pre>
Finally, to put things together we run this:
<pre>
cat newadj.txt | egrep -v '\#|\?|\!' | cut -f1 | perl -pe 's/ /<b\/>/g' > /tmp/bar4 && cat /tmp/bar3 | cut -f2 |
perl -pe 's/ /<b\/>/g' > /tmp/bar5
paste /tmp/bar4 /tmp/bar5 | perl -pe 's/<(\w+)>/<s n="$1"\/>/g' | awk -F'\t' '{ print " <e><l>"$1"</l><r>"$2"</r></e>" }' > apertium-bn-en.bn-en.dix.patch
</pre>
And we have our new entries in '''apertium-bn-en.bn-en.dix.patch'''.

Now, we need to check if the english lemmas that are entered in the new list, are actually present in the '''apertium-bn-en.en.dix'''. To do this, we can simply generate what we intend to add and what are already there. Then just check if ok:
<pre>
cat newadj.txt | egrep -v '\#|\?|\!' | cut -f2 > myadjs.txt
lt-expand apertium-bn-en.en.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | perl -pe 's/<.*//g' | python dev/uniq.py > existingadjs.txt
</pre>
Comparing these two files with a simple script there were 40 entries found those are not in "existingadjs.txt" thus not in '''apertium-bn-en.en.dix'''. We need to add those to get the transfer system working with the new entries.

@@ Line 72: / Line 72: @@
 grab the non-listed 100 nouns. And here's some of the output:
 <pre>
-বোধ<n><mf><nn>
+প্রজন্মের<n><mf><nn>
-ঝাপসা<n><nt><nn>
+ভরা<n><mf><nn>
-মেধা<n><mf><nn>
+সূত্র<n><nt><nn>
-গালমন্দ<n><nt><nn>
+নির্দেশ<n><mf><nn>
-পারিশ্রমিক<n><nt><nn>
 </pre>

@@ Line 70: / Line 70: @@
 </pre>
-grab the non-listed 100 nouns:
+grab the non-listed 100 nouns. And here's some of the output:
-<pre>
-diff noun.total noun.existing --normal | egrep -v '[0-9].*' | sed 's/< //g' | head -100 > noun.new100
-</pre>
-And here's some of the output:
 <pre>
 বোধ<n><mf><nn>

@@ Line 70: / Line 70: @@
 </pre>
+grab the non-listed 100 nouns:
-python code to grab unmatched entries (getnew.py):
 <pre>
+diff noun.total noun.existing --normal | egrep -v '[0-9].*' | sed 's/< //g' | head -100 > noun.new100
-import sys
-if len(sys.argv) != 4:
-	print 'wrong/no argument'
-	sys.exit();
-FILE1 = sys.argv[1]
-FILE2 = sys.argv[2]
-limit = int(sys.argv[3])
-li1,li2 = [],[]
-with open(FILE1,'r') as f1:
-	li1 = f1.readlines()
-with open(FILE2,'r') as f2:
-	li2 = f2.readlines()
-r = list(set(li1) - set(li2))
-if limit == -1 :
-	limit = len(r)
-for i in range(limit):
-	print r[i],
-</pre>
-to use this we just need to run:
-<pre>
-./getnew.py noun.total noun.existing 100
 </pre>
 And here's some of the output:
 <pre>
-ভরা<n><mf><nn>
+বোধ<n><mf><nn>
-সূত্র<n><nt><nn>
+ঝাপসা<n><nt><nn>
-নির্দেশ<n><mf><nn>
+মেধা<n><mf><nn>
-ধ্বংস<n><mf><nn>
+গালমন্দ<n><nt><nn>
-কারাদণ্ড<n><mf><nn>
+পারিশ্রমিক<n><nt><nn>
 </pre>

@@ Line 53: / Line 53: @@
 </pre>
 Comparing these two lists there were 40 entries found those are not in '''apertium-bn-en.en.dix'''. We need to add those to the english monodix to get the transfer system working with the new entries.
+== Answer : More (100 new nouns) ==
+grab the nouns from monodix:
+<pre>
+lt-expand apertium-bn-en.bn.dix| grep '<n>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | perl -pe 's/(<sg>|<pl>).*//g' | python dev/uniq.py > noun.total
+</pre>
+grab existing nouns in bdix:
+<pre>
+lt-expand apertium-bn-en.bn.dix| grep '<n>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | tee /tmp/foo1 | sed 's/^/^/g' | sed 's/$/$/g' |
+sed 's/$/^.<sent>$/g'  | apertium-pretransfer  | apertium-transfer apertium-bn-en.bn-en.t1x bn-en.t1x.bin bn-en.autobil.bin |
+apertium-interchunk apertium-bn-en.bn-en.t2x bn-en.t2x.bin | apertium-postchunk apertium-bn-en.bn-en.t3x bn-en.t3x.bin  | tee /tmp/foo2 |
+lt-proc -g bn-en.autogen.bin > /tmp/foo3 && paste /tmp/foo1 /tmp/foo2 /tmp/foo3 | egrep -v '\+' | egrep -v '@' | cut -f1 |
+perl -pe 's/(<sg>|<pl>).*//g' | python dev/uniq.py > noun.existing
+</pre>
+python code to grab unmatched entries (getnew.py):
+<pre>
+import sys
+if len(sys.argv) != 4:
+	print 'wrong/no argument'
+	sys.exit();
+FILE1 = sys.argv[1]
+FILE2 = sys.argv[2]
+limit = int(sys.argv[3])
+li1,li2 = [],[]
+with open(FILE1,'r') as f1:
+	li1 = f1.readlines()
+with open(FILE2,'r') as f2:
+	li2 = f2.readlines()
+r = list(set(li1) - set(li2))
+if limit == -1 :
+	limit = len(r)
+for i in range(limit):
+	print r[i],
+</pre>
+to use this we just need to run:
+<pre>
+./getnew.py noun.total noun.existing 100
+</pre>
+And here's some of the output:
+<pre>
+ভরা<n><mf><nn>
+সূত্র<n><nt><nn>
+নির্দেশ<n><mf><nn>
+ধ্বংস<n><mf><nn>
+কারাদণ্ড<n><mf><nn>
+</pre>

@@ Line 19: / Line 19: @@
 perl -pe 's/<comp>|<sup>//g' | python dev/uniq.py
 </pre>
-And we get, say #2 adjectives. Now, saving the adjective lists, we can run a simple script to find around 100 adjectives from those (#1-#2) non-listed entries. Say, we save these new adjectives into a file "newadj.txt". Now, we need to put the lemma mappings manually.
+And we get, say #2 adjectives. Now, saving the adjective lists, we can run a simple script to find around 100 adjectives from those (#1 - #2) non-listed entries. Say, we save these new adjectives into a file "newadj.txt". Now, we need to put the lemma mappings manually.
 A sample may be like this:
 <pre>

@@ Line 11: / Line 11: @@
 lt-expand apertium-bn-en.bn.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | perl -pe 's/<comp>|<sup>//g' | python dev/uniq.py
 </pre>
-There are 866 adjectives expanded. Now, to get the adjective list from the current system we can run this (according to the article mentioned above):
+Say there are #1 adjectives expanded. Now, to get the adjective list from the current system we can run this (according to the article mentioned above):
 <pre>
 lt-expand apertium-bn-en.bn.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | tee /tmp/foo1 | sed 's/^/^/g' | sed 's/$/$/g' | \
@@ Line 19: / Line 19: @@
 perl -pe 's/<comp>|<sup>//g' | python dev/uniq.py
 </pre>
-And we get 87 adjectives. Now, saving the adjective lists, we can run a simple script to find around 100 adjectives from those 866-87 = 779 non-listed entries. Say, we save these new adjectives into a file "newadj.txt". Now, we need to put the lemma mappings manually.
+And we get, say #2 adjectives. Now, saving the adjective lists, we can run a simple script to find around 100 adjectives from those (#1-#2) non-listed entries. Say, we save these new adjectives into a file "newadj.txt". Now, we need to put the lemma mappings manually.
 A sample may be like this:
 <pre>
@@ Line 27: / Line 27: @@
 ঝুঁকিপূর্ণ<adj><mf>	risky
 </pre>
-Now, for pos tagging we can again use the code from the mentioned article:
+Now, for pos-tagging we can again use the code from the mentioned article:
 <pre>
 cat newadj.txt | egrep -v '\#|\?|\!' | cut -f2 | tee /tmp/bar1 |  lt-proc en-bn.automorf.bin |  \
@@ Line 52: / Line 52: @@
 lt-expand apertium-bn-en.en.dix| grep '<adj>' | sed 's/:>:/:/g' | sed 's/:<:/:/g' | cut -f2 -d':' | perl -pe 's/<.*//g' | python dev/uniq.py > existingadjs.txt
 </pre>
-Comparing these two files with a simple script there were 40 entries found those are not in "existingadjs.txt" thus not in '''apertium-bn-en.en.dix'''. We need to add those to get the transfer system working with the new entries.
+Comparing these two lists there were 40 entries found those are not in '''apertium-bn-en.en.dix'''. We need to add those to the english monodix to get the transfer system working with the new entries.

@@ Line 25: / Line 25: @@
 তরল<adj><mf>	liquid
 মানসিক<adj><mf>	mental
-অসহায়<adj><mf>	helpless
+ঝুঁকিপূর্ণ<adj><mf>	risky
 </pre>
 Now, for pos tagging we can again use the code from the mentioned article:
@@ Line 41: / Line 41: @@
 And we have our new entries in '''apertium-bn-en.bn-en.dix.patch'''. Some samples:
 <pre>
-    <e><p><l>অজানা<s n="adj"/><s n="mf"/></l><r>unknown<s n="adj"/></r></p></e>
+<e><p><l>অজানা<s n="adj"/><s n="mf"/></l><r>unknown<s n="adj"/></r></p></e>
-    <e><p><l>তরল<s n="adj"/><s n="mf"/></l><r>liquid<s n="adj"/></r></p></e>
+<e><p><l>তরল<s n="adj"/><s n="mf"/></l><r>liquid<s n="adj"/></r></p></e>
-    <e><p><l>মানসিক<s n="adj"/><s n="mf"/></l><r>mental<s n="adj"/></r></p></e>
+<e><p><l>মানসিক<s n="adj"/><s n="mf"/></l><r>mental<s n="adj"/></r></p></e>
-    <e><p><l>অসহায়<s n="adj"/><s n="mf"/></l><r>helpless<s n="adj"/></r></p></e>
+<e><p><l>ঝুঁকিপূর্ণ<s n="adj"/><s n="mf"/></l><r>risky<s n="adj"/><s n="sint"/></r></p></e>
 </pre>