Automatically generating compound bidix entries - Revision history

Bech: Category:Documentation in English

2012-03-24T11:28:07Z

Category:Documentation in English

Francis Tyers at 21:38, 11 March 2010

2010-03-11T21:38:21Z

Francis Tyers at 21:32, 11 March 2010

2010-03-11T21:32:40Z

Francis Tyers at 21:30, 11 March 2010

2010-03-11T21:30:12Z

Francis Tyers: /* Perspectives for improvement */

2010-03-10T00:14:09Z

Perspectives for improvement

Francis Tyers: /* Perspectives for improvement */

2010-03-10T00:10:14Z

Perspectives for improvement

Francis Tyers: /* Perspectives for improvement */

2010-03-10T00:06:09Z

Perspectives for improvement

Francis Tyers: /* Perspectives for improvement */

2010-03-10T00:03:35Z

Perspectives for improvement

Francis Tyers at 14:41, 9 March 2010

2010-03-09T14:41:46Z

Francis Tyers: /* Score with LM */

2010-03-09T14:36:53Z

Score with LM

← Older revision		Revision as of 11:28, 24 March 2012
Line 227:		Line 227:

	[[Category:Development]]		[[Category:Development]]
			[[Category:Documentation in English]]

@@ Line 5: / Line 5: @@
 * We have an existing translator, with a language pair where the source side has productive compounding.
 * The source dictionary and bilingual dictionary are not so big, so we miss frequent words, and we also miss compounds
-* We want to be able to translate compounds, but not necessarily ''on-line'' because of the high failure rate, due to either disambiguation or transfer, or just incomplete entries.
+* We want to be able to translate compounds, but not necessarily ''on-line'' because of the high failure rate, due to either disambiguation or transfer, just incomplete entries.
 * We can try and generate compound entries for bilingual dictionary and monolingual target language dictionary, ''offline'', proofread them and include them just as normal.
+* This process should be automated as much as possible.
 ==Process==

@@ Line 2: / Line 2: @@
 ==Rationale==
+* We have an existing translator, with a language pair where the source side has productive compounding.
+* The source dictionary and bilingual dictionary are not so big, so we miss frequent words, and we also miss compounds
+* We want to be able to translate compounds, but not necessarily ''on-line'' because of the high failure rate, due to either disambiguation or transfer, or just incomplete entries.
+* We can try and generate compound entries for bilingual dictionary and monolingual target language dictionary, ''offline'', proofread them and include them just as normal.
 ==Process==

@@ Line 1: / Line 1: @@
 {{TOCD}}
-==Likely compound wordlist==
+==Rationale==
+==Process==
+===Likely compound wordlist===
 Ideas: Remove words that have >1 capital letter, remove words with < 7 letters.
@@ Line 18: / Line 23: @@
 </pre>
-==Decompound them with lttoolbox==
+===Decompound them with lttoolbox===
 <pre>
@@ Line 35: / Line 40: @@
 </pre>
-==Remove words which still don't have analyses==
+===Remove words which still don't have analyses===
 <pre>
@@ Line 41: / Line 46: @@
 </pre>
-==Generate correspondences==
+===Generate correspondences===
 Between original surface form and analyses:
@@ Line 84: / Line 89: @@
 </pre>
-==Translate==
+===Translate===
 Run the file you just made  through the rest of the pipeline:
@@ Line 108: / Line 113: @@
 </pre>
-==Score with LM==
+===Score with LM===
 <pre>
@@ Line 141: / Line 146: @@
 </pre>
-==Convert to bidix==
+===Convert to bidix===
 <pre>
@@ Line 203: / Line 208: @@
 </pre>
-==Perspectives for improvement==
+===Perspectives for improvement===
 * Removing impossible combinations (e.g. <code><dat> + <nom></code> or <code><pr> + <nom></code> compounds)

@@ Line 213: / Line 213: @@
 * Alternative decompounding strategies, e.g.
 ** We cannot decompound "þýskumælandi" 'German speaker' with {{sc|lrlm}} because "þýskum" is a dative form of 'German' and the longest match.
+* Read the bilingual dictionary and generate all possible <code>slr</code> entries.
 [[Category:Development]]

@@ Line 209: / Line 209: @@
 ** In {{sc|nom nom}} compounds, the first noun must always be singular in English, e.g.
 *** vörumerkjasafn → <nowiki>*</nowiki>trademarks museum → trademark museum
+* Post-processing to merge equivalent entries, e.g.
+** If we have two entries, one for singular and one for plural, then they should be merged.
 [[Category:Development]]

@@ Line 139: / Line 139: @@
 -0.999014	||	sport articles .	[íþróttgreinar].[]	^íþrótt<n><f><sg><nom><ind>+greinir<n><m><pl><nom><ind>$
 -0.999014	||	sport articles .	[íþróttgreinar].[]	^íþrótt<n><f><sg><dat><ind>+grein<n><f><pl><nom><ind>$
+</pre>
+==Convert to bidix==
+<pre>
+#!/usr/bin/python2.5
+# coding=utf-8
+# -*- encoding: utf-8 -*-
+import sys, codecs, copy, re, commands, os;
+sys.stdin = codecs.getreader('utf-8')(sys.stdin);
+sys.stdout = codecs.getwriter('utf-8')(sys.stdout);
+sys.stderr = codecs.getwriter('utf-8')(sys.stderr);
+#-0.999014	||	sport articles .	[íþróttgreinar].[]	^íþrótt<n><f><sg><acc><ind>+grein<n><f><pl><nom><ind>$
+def generate(s): #{
+	gen = '^' + s + '$';
+	cmd = 'echo "' + gen + '" | lt-proc -g ' + sys.argv[2];
+	return commands.getstatusoutput(cmd)[1];
+#}
+def tags_to_bidix(t): #{
+	pos = t[0].strip('><');
+	gender = t[1].strip('><');
+	bdtags = '<s n="' + pos + '"/><s n="' + gender + '"/>';
+	return bdtags;
+#}
+for line in file(sys.argv[1]).read().split('\n'): #{
+	row = line.split('\t');
+	prob = row[0];
+	translation = row[2].strip('. ');
+	analysis = row[4].strip('^$');
+	analysis_row = analysis.split('+');
+	head = analysis_row[len(analysis_row)-1];
+	queue = analysis_row[0:-1];
+	q = '';
+	for part in queue: #{
+		q = q + generate(part);
+	#}
+	#print prob , queue , q + head , translation;
+	tags = head.split('<')[1:];
+	lemma = head.split('<')[0];
+	left = q + lemma + tags_to_bidix(tags);
+	print '<!-- p: ' + str(prob) + ' --> <e><p><l>' + left + '</l><r>' + translation.replace(' ', '<b/>') + '<s n="n"/></r></p></e>';
+#}
+</pre>
+<pre>
+$ python sorted-to-bidix.py unk.trans.sorted.txt ../is.gen.bin > unk.bidix
+$ cat unk.bidix  | sort -ur
+<!-- p: -0.999912 --> <e><p><l>tollhlið<s n="n"/><s n="nt"/></l><r>customs<b/>gate<s n="n"/></r></p></e>
+<!-- p: -0.999877 --> <e><p><l>starfsmannaleiga<s n="n"/><s n="f"/></l><r>employees<b/>rents<s n="n"/></r></p></e>
+<!-- p: -0.999793 --> <e><p><l>heilsuátak<s n="n"/><s n="nt"/></l><r>health<b/>effort<s n="n"/></r></p></e>
+<!-- p: -0.999014 --> <e><p><l>viðskiptagrein<s n="n"/><s n="f"/></l><r>businesses<b/>articles<s n="n"/></r></p></e>
+<!-- p: -0.999014 --> <e><p><l>viðskiptagreinir<s n="n"/><s n="m"/></l><r>businesses<b/>articles<s n="n"/></r></p></e>
+<!-- p: -0.999014 --> <e><p><l>íþróttgrein<s n="n"/><s n="f"/></l><r>sport<b/>articles<s n="n"/></r></p></e>
 </pre>