Building a pseudo-parallel corpus - Revision history

Fpetkovski: /* Example */

2012-08-23T23:25:24Z

Example

Fpetkovski at 23:13, 23 August 2012

2012-08-23T23:13:09Z

Fpetkovski: /* Building a pseudo-parallel corpus */

2012-08-23T22:39:50Z

Building a pseudo-parallel corpus

Fpetkovski at 22:39, 23 August 2012

2012-08-23T22:39:08Z

Fpetkovski: /* Building a pseudo-parallel corpus */

2012-08-23T21:50:27Z

Building a pseudo-parallel corpus

Francis Tyers at 10:50, 23 August 2012

2012-08-23T10:50:40Z

Francis Tyers at 10:50, 23 August 2012

2012-08-23T10:50:14Z

Francis Tyers at 10:49, 23 August 2012

2012-08-23T10:49:36Z

Fpetkovski at 01:57, 23 August 2012

2012-08-23T01:57:39Z

Fpetkovski at 19:59, 22 August 2012

2012-08-22T19:59:43Z

@@ Line 23: / Line 23: @@
 <pre>
-# cat testing-set | biltrans | put a full stop at the end of each sentence | select default
+# extract defaults
-echo "generating biltrans..."
-cat testing-set | apertium -d ~/Apertium/apertium-mk-en mk-en-biltrans | apertium-ldx-proc /tmp/mken-defaults.bin > testing-set-biltrans;
+ ~/Apertium/apertium-lex-tools/scripts/extract-default-ldx.py ~/Apertium/apertium-mk-en/apertium-mk-en.mk-en.dix lr | grep -v '"pr"' > mken-defaults.dix
+lt-comp lr mken-defaults.dix mken-defaults.bin
+# cat testing-set | generate biltrans | select default
+cat testing-set | apertium -d ~/Apertium/apertium-mk-en mk-en-biltrans | apertium-ldx-proc mken-defaults.bin > testing-set-biltrans;
 # expand
-echo "expand..."
 cat testing-set-biltrans | ~/Apertium/apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py > testing-set-biltrans-expanded
-echo "done"
-echo "generate sl side..."
-# generate sl-side
-cat testing-set-biltrans-expanded | sed 's/\(\^[^\/]*\)[\/][^\$]*\$/\1$/g' > sl-testing-5grams
-echo "done"
-echo "generate tl-side..."
 # generate tl-side
-cat testing-set-biltrans-expanded | sed 's/~/ /g' | sed 's/$/^.<sent>$/g' | bash biltrans-to-end.sh > tl-testing-5grams
+cat testing-set-biltrans-expanded | bash biltrans-to-end.sh > testing-set-expanded
-echo "done"
+#train a language model
+build-lm.sh -i training-set -o lm.lm.gz -n 5 -b
+compile-lm lm.lm.gz lmodel.bin
+cat testing-set-expanded | ~/Apertium/apertium-lex-learner/irstlm_ranker_max lmodel.bin | grep "|@|" > testing-set-final
 </pre>

@@ Line 11: / Line 11: @@
 * run the corpus through mk-en-biltrans
-* Run through apertium-lex-tools/scripts/extract-default-ldx.py to select the defaults for words with a POS different from <pr>. This step is necessary in order not to get an explosion of the possible TL sentences.
+* Run through apertium-ldx-proc to select the defaults for words with a POS different from <pr>. This step is necessary in order not to get an explosion of the possible TL sentences.
 * Run through apertium-lex-tools/scripts/biltrans-to-multitrans.py to expand the biltrans sentence to cover all the possible lexical transfers. You can also use apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py, a slower version but it uses less memory.
 * Run through the rest of the pipeline from apertium-transfer -b onwards to get target-language sentences
@@ Line 18: / Line 18: @@
 This way for each expanded translation you will get source-language to target-language probability for each SL:TL pair. The most probable translation will be marked with |@|.
+== Example ==
+An example script for generating a mk-en pseudo-parallel corpus:
+<pre>
+# cat testing-set | biltrans | put a full stop at the end of each sentence | select default
+echo "generating biltrans..."
+cat testing-set | apertium -d ~/Apertium/apertium-mk-en mk-en-biltrans | apertium-ldx-proc /tmp/mken-defaults.bin > testing-set-biltrans;
+# expand
+echo "expand..."
+cat testing-set-biltrans | ~/Apertium/apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py > testing-set-biltrans-expanded
+echo "done"
+echo "generate sl side..."
+# generate sl-side
+cat testing-set-biltrans-expanded | sed 's/\(\^[^\/]*\)[\/][^\$]*\$/\1$/g' > sl-testing-5grams
+echo "done"
+echo "generate tl-side..."
+# generate tl-side
+cat testing-set-biltrans-expanded | sed 's/~/ /g' | sed 's/$/^.<sent>$/g' | bash biltrans-to-end.sh > tl-testing-5grams
+echo "done"
+</pre>

@@ Line 12: / Line 12: @@
 * run the corpus through mk-en-biltrans
 * Run through apertium-lex-tools/scripts/extract-default-ldx.py to select the defaults for words with a POS different from <pr>. This step is necessary in order not to get an explosion of the possible TL sentences.
-* Run through apertium-lex-tools/scripts/biltrans-to-multitrans.py to expand the biltrans sentence to cover all the possible lexical transfers. You can also use apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py, a slower version but uses less memory.
+* Run through apertium-lex-tools/scripts/biltrans-to-multitrans.py to expand the biltrans sentence to cover all the possible lexical transfers. You can also use apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py, a slower version but it uses less memory.
 * Run through the rest of the pipeline from apertium-transfer -b onwards to get target-language sentences
 * Run through apertium-lex-learner/irstlm-ranker-max
 This way for each expanded translation you will get source-language to target-language probability for each SL:TL pair. The most probable translation will be marked with |@|.

@@ Line 3: / Line 3: @@
 == IRSTLM ==
 IRSTLM is a tool for building n-gram language models from corpora. It supports different smoothing and interpolation methods, including Written-Bell smoothing, Kneser-Ney smoothing and others.
+biltrans-to-multitrans-recu.py
 The full documentation can be viewed [http://sourceforge.net/apps/mediawiki/irstlm/index.php?title=Main_Page here], and the whole toolkit can be downloaded [http://sourceforge.net/projects/irstlm/ here]
@@ Line 11: / Line 11: @@
 * run the corpus through mk-en-biltrans
-* Run through <code>apertium-lex-tools/scripts/biltrans-to-multitrans.py</code>
+* Run through apertium-lex-tools/scripts/extract-default-ldx.py to select the defaults for words with a POS different from <pr>. This step is necessary in order not to get an explosion of the possible TL sentences.
+* Run through apertium-lex-tools/scripts/biltrans-to-multitrans.py to expand the biltrans sentence to cover all the possible lexical transfers. You can also use apertium-lex-tools/scripts/biltrans-to-multitrans-line-recursive.py, a slower version but uses less memory.
-* Run through the rest of the pipeline from apertium-transfer -b onwards
-* Run through <code>apertium-lex-learner/irstlm-ranker<code>
+* Run through the rest of the pipeline from apertium-transfer -b onwards to get target-language sentences
+* Run through apertium-lex-learner/irstlm-ranker-max
+This way for each expanded translation you will get source-language to target-language probability for each SL:TL pair. The most probable translation will be marked with |@|.

@@ Line 10: / Line 10: @@
 The main idea is to get a source-language corpus and run it through the apertium pipeline, but this time let the language model choose the preposition instead of apertium. The main algorithm is as follows (example for mk-en):
+* run the corpus through mk-en-biltrans
-<pre>
-run the corpus through mk-en-biltrans
+* Run through <code>apertium-lex-tools/scripts/biltrans-to-multitrans.py</code>
-Run through <code>apertium-lex-tools/scripts/biltrans-to-multitrans.py</code>
+* Run through the rest of the pipeline from apertium-transfer -b onwards
-Run through the rest of the pipeline from apertium-transfer -b onwards
+* Run through <code>apertium-lex-learner/irstlm-ranker<code>
-Run through <code>apertium-lex-learner/irstlm-ranker<code>
-</pre>

← Older revision		Revision as of 10:50, 23 August 2012
Line 17:		Line 17:

	</pre>		</pre>

	optimization

← Older revision		Revision as of 10:50, 23 August 2012
Line 18:		Line 18:
	</pre>		</pre>

	optimization		optimization

← Older revision		Revision as of 10:49, 23 August 2012
Line 17:		Line 17:

	</pre>		</pre>

			optimization