Running the MaxEnt rule learning - Revision history

Francis Tyers at 21:25, 14 February 2014

2014-02-14T21:25:06Z

Fpetkovski: /* Bilingual rule learning */

2013-06-26T10:45:28Z

Bilingual rule learning

Fpetkovski: /* Bilingual rule learning */

2013-06-26T10:44:37Z

Bilingual rule learning

Fpetkovski: /* Monolingual rule learning */

2013-06-26T10:44:23Z

Monolingual rule learning

Fpetkovski: /* Bilingual rule learning */

2013-06-26T10:43:48Z

Bilingual rule learning

Fpetkovski at 13:31, 17 June 2013

2013-06-17T13:31:00Z

Fpetkovski: /* Monolingual rule learning */

2013-06-17T13:28:36Z

Monolingual rule learning

Fpetkovski: /* Monolingual rule learning */

2013-06-17T13:28:08Z

Monolingual rule learning

Fpetkovski: /* Monolingual rule learning */

2013-06-17T10:45:37Z

Monolingual rule learning

Fpetkovski: /* Monolingual rule learning */

2013-06-17T10:31:40Z

Monolingual rule learning

← Older revision		Revision as of 21:25, 14 February 2014
Line 86:		Line 86:

	The variables should be set the same way as with the monolingual method.		The variables should be set the same way as with the monolingual method.

			[[Category:Lexical selection]]

@@ Line 63: / Line 63: @@
 MIN=$1
-python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
 python $SCRIPTS/ngram-count-patterns-maxent.py $DATA/setimes.sh-mk.freq $TRAIN/$CORPUS.candidates.hr-mk 2>ngrams > events
 echo -n "" > all-lambdas

@@ Line 63: / Line 63: @@
 MIN=$1
-# python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
+python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
 python $SCRIPTS/ngram-count-patterns-maxent.py $DATA/setimes.sh-mk.freq $TRAIN/$CORPUS.candidates.hr-mk 2>ngrams > events
 echo -n "" > all-lambdas

@@ Line 16: / Line 16: @@
 MIN=$1
-# python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
+python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
 echo -n "" > all-lambdas
 cat events | grep -v -e '\$ 0\.0 #' -e '\$ 0 #' > events.trimmed

@@ Line 45: / Line 45: @@
 ==Bilingual rule learning==
+You will need to run [[Generating lexical-selection rules from a parallel corpus]] first in order
+to obtain the (corpus).candidates.(lang-pair) file (You don't have to run the whole process. Stop when
+you have obtained the candidates file).
+You also need to have yasmet compiled in your apertium-lex-tools directory.
+Next, run the following script to obtain the lexical selection rules
+<pre>
+YASMET=/home/philip/Apertium/apertium-lex-tools/yasmet
+SCRIPTS=/home/philip/Apertium/apertium-lex-tools/scripts
+TRAIN=/home/philip/Apertium/corpora/raw/setimes-hr-mk-nikola
+DATA=/home/philip/Apertium/gsoc2013/monolingual/data
+CORPUS=setimes
+PAIR=sh-mk
+MIN=$1
+# python3 $SCRIPTS/biltrans-count-patterns-frac-maxent.py $DATA/setimes.sh-mk.freq $DATA/setimes.sh-mk.ambig $DATA/setimes.sh-mk.annotated > events 2>ngrams
+python $SCRIPTS/ngram-count-patterns-maxent.py $DATA/setimes.sh-mk.freq $TRAIN/$CORPUS.candidates.hr-mk 2>ngrams > events
+echo -n "" > all-lambdas
+cat events | grep -v -e '\$ 0\.0 #' -e '\$ 0 #' > events.trimmed
+for i in `cat events.trimmed|cut -f1 |sort -u`; do
+	num=`cat events.trimmed | grep "^$i" | cut -f2 | head -1`
+	echo $num > tmp.yasmet;
+	cat events.trimmed | grep "^$i" | cut -f3 | sed 's/^ //g' | sed 's/[ ]*$//g'  >> tmp.yasmet;
+	cat tmp.yasmet | $YASMET -red $MIN > tmp.yasmet.$MIN;
+	cat tmp.yasmet.$MIN | $YASMET > tmp.lambdas
+	cat tmp.lambdas | sed "s/^/$i /g" >> all-lambdas;
+done
+rm tmp.*
+python3 $SCRIPTS/merge-ngrams-lambdas.py ngrams all-lambdas > rules-all.txt
+python3 $SCRIPTS/lambdas-to-rules.py $DATA/setimes.sh-mk.freq rules-all.txt > ngrams-all.txt
+python3 $SCRIPTS/ngrams-to-rules-me.py ngrams-all.txt > $PAIR.ngrams-lm-$MIN.xml 2>/tmp/$PAIR.$MIN.lög
+</pre>
+The variables should be set the same way as with the monolingual method.

@@ Line 43: / Line 43: @@
  * CORPUS is the base name of the corpus file
  * PAIR is the language pair
+==Bilingual rule learning==

@@ Line 4: / Line 4: @@
 First run the [[monolingual-rule-learning]] to obtain the .ambig and .annotated files.
-You you should also have yasmet compiled in your Apertium-lex-tools folder.
+You should also have yasmet compiled in your Apertium-lex-tools folder.
 Next, run the following script to extract lexical selection rules:

← Older revision		Revision as of 10:31, 17 June 2013
Line 2:		Line 2:

	==Monolingual rule learning==		==Monolingual rule learning==

			First run the [[Monolingual rule learning]]