Running the monolingual rule learning - Revision history

Memduh at 19:01, 17 August 2018

2018-08-17T19:01:41Z

Memduh at 18:38, 17 August 2018

2018-08-17T18:38:27Z

Francis Tyers at 21:25, 14 February 2014

2014-02-14T21:25:13Z

Fpetkovski at 01:15, 16 July 2013

2013-07-16T01:15:13Z

Fpetkovski at 12:41, 23 June 2013

2013-06-23T12:41:04Z

Fpetkovski at 05:54, 23 June 2013

2013-06-23T05:54:08Z

Fpetkovski at 22:04, 14 June 2013

2013-06-14T22:04:36Z

Fpetkovski at 22:01, 14 June 2013

2013-06-14T22:01:40Z

Fpetkovski at 21:57, 14 June 2013

2013-06-14T21:57:54Z

Fpetkovski at 21:57, 14 June 2013

2013-06-14T21:57:34Z

@@ Line 1: / Line 1: @@
 Prerequisites:
 * Install [[apertium-lex-tools]]
-* Install IRSTLM (http://sourceforge.net/projects/irstlm/)
+* Install IRSTLM (https://github.com/douglasbagnall/irstlm)
 * Train a target side language model (http://hermes.fbk.eu/people/bertoldi/teaching/lab_2010-2011/img/irstlm-manual.pdf)
 * The language pair must support the pretransfer and multi modes. See apertium-sh-mk/modes.xml

@@ Line 2: / Line 2: @@
 * Install [[apertium-lex-tools]]
 * Install IRSTLM (http://sourceforge.net/projects/irstlm/)
-* Train a target side language model (http://sourceforge.net/apps/mediawiki/irstlm/index.php?title=User_Manual)
+* Train a target side language model (http://hermes.fbk.eu/people/bertoldi/teaching/lab_2010-2011/img/irstlm-manual.pdf)
 * The language pair must support the pretransfer and multi modes. See apertium-sh-mk/modes.xml
 as a reference on how to add these modes if they do not exist.

← Older revision		Revision as of 21:25, 14 February 2014
Line 67:		Line 67:

	Finally, executing the Makefile will generate lexical selection rules for the specified language pair.		Finally, executing the Makefile will generate lexical selection rules for the specified language pair.

			[[Category:Lexical selection]]

@@ Line 5: / Line 5: @@
 * The language pair must support the pretransfer and multi modes. See apertium-sh-mk/modes.xml
 as a reference on how to add these modes if they do not exist.
+Place the following Makefile in the folder where you want to run your training process:
 <pre>
 CORPUS=setimes
-Place the following Makefile in the folder where you want to run your training process:
 DIR=sh-mk
 DATA=/home/philip/Apertium/apertium-sh-mk/
@@ Line 15: / Line 14: @@
 SCRIPTS=/home/philip/Apertium/apertium-lex-tools/scripts
 MODEL=/home/philip/Apertium/corpora/language-models/mk/setimes.mk.5.blm
+LEX_TOOLS=/home/philip/Apertium/apertium-lex-tools
+THR=0
 #all: data/$(CORPUS).$(DIR).lrx data/$(CORPUS).$(DIR).freq.lrx
-all: data/$(CORPUS).$(DIR).freq.lrx.bin
+all: data/$(CORPUS).$(DIR).freq.lrx.bin data/$(CORPUS).$(DIR).patterns.lrx
-data/$(CORPUS).$(DIR).lines: $(CORPUS).$(DIR).txt
+data/$(CORPUS).$(DIR).tagger: $(CORPUS).$(DIR).txt
 	if [ ! -d data ]; then mkdir data; fi
+	cat $(CORPUS).$(DIR).txt | sed 's/[^\.]$$/./g' | apertium-destxt | apertium -f none -d $(DATA) $(DIR)-tagger | apertium-pretransfer > $@
-	seq `cat $< | wc -l` > $@
-data/$(CORPUS).$(DIR).biltrans: $(CORPUS).$(DIR).txt
+data/$(CORPUS).$(DIR).ambig: data/$(CORPUS).$(DIR).tagger
+	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/multitrans $(DATA)$(DIR).autobil.bin -b -t > $@
-	if [ ! -d data ]; then mkdir data; fi
-	cat $(CORPUS).$(DIR).txt | apertium-destxt | apertium -f none -d $(DATA) $(DIR)-pretransfer | lt-proc -b $(DATA)/$(AUTOBIL) > $@
+data/$(CORPUS).$(DIR).multi-trimmed: data/$(CORPUS).$(DIR).tagger
-data/$(CORPUS).$(DIR).ambig: data/$(CORPUS).$(DIR).biltrans data/$(CORPUS).$(DIR).lines
+	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/multitrans $(DATA)$(DIR).autobil.bin -m -t > $@
-	cat -n data/$(CORPUS).$(DIR).biltrans | python3 $(SCRIPTS)/trim-fertile-lines.py | python3 $(SCRIPTS)/biltrans-line-only-pos-ambig.py | python3 $(SCRIPTS)/biltrans-trim-uncovered.py > $@
-data/$(CORPUS).$(DIR).multi: data/$(CORPUS).$(DIR).ambig
-	cat $< | python $(SCRIPTS)/biltrans-to-multitrans-line-recursive.py > $@
-data/$(CORPUS).$(DIR).unranked: data/$(CORPUS).$(DIR).multi
-	cat $< | apertium -f none -d $(DATA) $(DIR)-multi > $@
-data/$(CORPUS).$(DIR).ranked: data/$(CORPUS).$(DIR).unranked
-	cat $< | irstlm-ranker-frac $(MODEL) > $@
-data/$(CORPUS).$(DIR).annotated: data/$(CORPUS).$(DIR).multi data/$(CORPUS).$(DIR).ranked
+data/$(CORPUS).$(DIR).ranked: data/$(CORPUS).$(DIR).tagger
-	paste data/$(CORPUS).$(DIR).multi data/$(CORPUS).$(DIR).ranked | cut -f1-4 > $@
+	cat $< | $(LEX_TOOLS)/multitrans $(DATA)$(DIR).autobil.bin -m | apertium -f none -d $(DATA) $(DIR)-multi | irstlm-ranker-frac $(MODEL) > $@
+data/$(CORPUS).$(DIR).annotated: data/$(CORPUS).$(DIR).multi-trimmed data/$(CORPUS).$(DIR).ranked
+	paste data/$(CORPUS).$(DIR).multi-trimmed data/$(CORPUS).$(DIR).ranked | cut -f1-4 > $@
 data/$(CORPUS).$(DIR).freq: data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated
 	python3 $(SCRIPTS)/biltrans-extract-frac-freq.py  data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated > $@
+data/$(CORPUS).$(DIR).freq.lrx:  data/$(CORPUS).$(DIR).freq
+	python3 $(SCRIPTS)/extract-alig-lrx.py $< > $@
+data/$(CORPUS).$(DIR).freq.lrx.bin: data/$(CORPUS).$(DIR).freq.lrx
+	lrx-comp $< $@
 data/$(CORPUS).$(DIR).ngrams: data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated
 	python3 $(SCRIPTS)/biltrans-count-patterns-ngrams.py data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated > $@
@@ Line 51: / Line 51: @@
 	python3 $(SCRIPTS)/ngram-pruning-frac.py data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ngrams > $@
-data/$(CORPUS).$(DIR).freq.lrx:  data/$(CORPUS).$(DIR).freq
+data/$(CORPUS).$(DIR).patterns.lrx:  data/$(CORPUS).$(DIR).patterns
-	python3 $(SCRIPTS)/extract-alig-lrx.py $< > $@
+	python3 $(SCRIPTS)/ngrams-to-rules.py $< $(THR) > $@
-data/$(CORPUS).$(DIR).freq.lrx.bin: data/$(CORPUS).$(DIR).freq.lrx
-	apertium-lrx-comp $< $@
 </pre>

@@ Line 5: / Line 5: @@
 * The language pair must support the pretransfer and multi modes. See apertium-sh-mk/modes.xml
 as a reference on how to add these modes if they do not exist.
-Place the following Makefile in the folder where you want to run your training process:
 <pre>

@@ Line 62: / Line 62: @@
 In the same folder also place your source side corpus file. The corpus file needs to be named as "basename"."language-pair".txt. <br/>
 As an illustration, in the Makefile example, the corpus file is named setimes.sh-mk.txt.
 Set the Makefile variables as follows: <br/>
@@ Line 73: / Line 70: @@
 * SCRIPTS denotes the path to the lex-tools scripts
 * MODEL is the path to the target side (binary) language model used for scoring the possible translations of ambiguous words
+Finally, executing the Makefile will generate lexical selection rules for the specified language pair.

← Older revision		Revision as of 21:57, 14 June 2013
Line 61:		Line 61:

	In the same folder also place your source side corpus file. The corpus file needs to be named as "basename"."language-pair".txt. <br/>		In the same folder also place your source side corpus file. The corpus file needs to be named as "basename"."language-pair".txt. <br/>
	In the Makefile example, the corpus file is named setimes.sh-mk.txt.		As an illustration, in the Makefile example, the corpus file is named setimes.sh-mk.txt.