Partial hack for prefix inflection - Revision history

Francis Tyers at 18:47, 15 April 2012

2012-04-15T18:47:09Z

Bech: Category:Documentation in English

2012-03-24T11:44:04Z

Category:Documentation in English

Francis Tyers: /* Pretagger */

2007-12-21T14:39:37Z

Pretagger

Francis Tyers at 23:02, 20 December 2007

2007-12-20T23:02:49Z

Francis Tyers: /* Pipeline */

2007-12-20T23:01:30Z

Pipeline

Francis Tyers: /* Analysis */

2007-12-20T23:01:08Z

Analysis

Francis Tyers at 22:56, 20 December 2007

2007-12-20T22:56:29Z

Francis Tyers: /* Pretagger */

2007-12-20T22:53:26Z

Pretagger

Francis Tyers at 22:37, 20 December 2007

2007-12-20T22:37:55Z

Francis Tyers at 22:35, 20 December 2007

2007-12-20T22:35:42Z

@@ Line 1: / Line 1: @@
+{{deprecated}}
 {{TOCD}}
 This is a description of a partial hack for dealing with prefix inflection (see [[prefixes and infixes]]). Essentially it re-uses as much as possible components that we already have, although requires one more component, a <code>pretagger</code> module.

← Older revision		Revision as of 11:44, 24 March 2012
Line 261:		Line 261:

	[[Category:Development]]		[[Category:Development]]
			[[Category:Documentation in English]]

@@ Line 155: / Line 155: @@
 reading = '';
-def reverse_order(tags, out): #{
+def pop_tags(tags, out): #{
 	if len(tags) < 1: #{
 		return out;
 	#}
 	out = out + tags.pop();
-	return reverse_order(tags, out);
+	return pop_tags(tags, out);
+#}
+def pull_tags(tags, out): #{
+	for tag in tags: #{
+		out = out + tag;
+	#}
+	return out;
 #}
 for char in sys.stdin.read().decode('utf-8'): #{
 	if char == '^': #{
+		sys.stdout.write(char);
 		reading = 'form';
 		continue;
@@ Line 171: / Line 178: @@
 	if char == '/': #{
 		reading = 'lemma';
+		if len(form) < 1 and len(lemma) > 1: #{
+			sys.stdout.write(lemma + pop_tags(tags, ''));
+			lemma = '';
+			tags = [];
+		#}
+		sys.stdout.write(form);
+		sys.stdout.write(char);
+		form = '';
+		continue;
+	#}
+	if char == '#': #{
+		reading = 'pass';
+		if len(form) < 1 and len(lemma) > 1: #{
+			sys.stdout.write(lemma + pop_tags(tags, ''));
+			form = '';
+			lemma = '';
+			tags = [];
+		#}
+		sys.stdout.write(char);
+		continue;
+	#}
+	if char == '+': #{
+		reading = 'pass';
+		if len(form) < 1 and len(lemma) > 1: #{
+			sys.stdout.write(lemma + pull_tags(tags, ''));
+			form = '';
+			lemma = '';
+			tags = [];
+		#}
+		sys.stdout.write(char);
 		continue;
 	#}
-	if char == '<': #{
+	if char == '<' and reading is not 'pass': #{
 		tag = '';
 		reading = 'tag';
 	#}
-	if char == '>': #{
+	if char == '>' and reading is not 'pass': #{
 		tag = tag + char;
 		tags.append(tag);
@@ Line 184: / Line 227: @@
 	#}
 	if char == '$': #{
+		if len(form) < 1 and len(lemma) > 1: #{
-		reading = '';
+			sys.stdout.write(lemma + pop_tags(tags, ''));
+			reading = 'lemma';
+		#}
+		tags = [];
+		form = '';
+		lemma = '';
+		sys.stdout.write(char);
+		reading = 'pass';
 		continue;
-	#}
-	if char == ' ' or char == '\n': #{
-		reading = 'output';
 	#}
 	if reading == 'form': #{
 		form = form + char;
+		continue;
 	#}
 	if reading == 'lemma': #{
 		lemma = lemma + char;
+		continue;
 	#}
 	if reading == 'tag': #{
 		tag = tag + char;
+		continue;
 	#}
-	if reading == 'output': #{
+	if reading == 'pass': #{
-		sys.stdout.write('^' + form + '/' + lemma + reverse_order(tags, '') + '$');
-		tags = [];
-		tag = '';
-		lemma = '';
-		form = '';
 		sys.stdout.write(char);
+		continue;
 	#}
 #}
 </pre>

@@ Line 2: / Line 2: @@
 This is a description of a partial hack for dealing with prefix inflection (see [[prefixes and infixes]]). Essentially it re-uses as much as possible components that we already have, although requires one more component, a <code>pretagger</code> module.
-The first dictionary would be written in the normal manner, the second dictionary would be auto-generated from the first by means of lt-expand output coupled with the pretagger output.
+The first dictionary would be written in the normal manner, the second dictionary would be auto-generated from the first by means of <code>lt-expand</code> output coupled with the pretagger output.
 Note: This is basically a hack on Mikel's suggestion #1 on the [[prefixes and infixes]] page.

@@ Line 13: / Line 13: @@
 $ echo "watu mtu" | lt-proc sw.automorf.bin
 ^watu/m<pl>tu<n>$ ^mtu/m<sg>tu<n>$
-$ echo "watu mtu" | lt-proc sw.automorf.bin  | python pretagger.py
+$ echo "watu mtu" | lt-proc sw.automorf.bin  | pretagger.py
 ^watu/mtu<n><pl>$ ^mtu/mtu<n><sg>$
-$ echo "watu mtu" | lt-proc sw.automorf.bin  | python pretagger.py | apertium-tagger -g sw.prob
+$ echo "watu mtu" | lt-proc sw.automorf.bin  | pretagger.py | apertium-tagger -g sw.prob
 ^mtu<n><pl>$ ^mtu<n><sg>$
-$ echo "watu mtu" | lt-proc sw.automorf.bin  | python pretagger.py | apertium-tagger -g sw.prob  | lt-proc -g sw.automangle.bin
+$ echo "watu mtu" | lt-proc sw.automorf.bin  | pretagger.py | apertium-tagger -g sw.prob  | lt-proc -g sw.automangle.bin
 m<pl>tu<n> m<sg>tu<n>
 </pre>

@@ Line 5: / Line 5: @@
 Note: This is basically a hack on Mikel's suggestion #1 on the [[prefixes and infixes]] page.
+One of the advantages of this system is that it is that the "normalisation" scheme (dictionary 2) can be generated from the dictionary + the pretagger and doesn't need to be hand-coded. Disadvantages are that it adds another layer, and probably misses a lot of nuances (e.g. multiwords, etc.).
 ==Analysis==