View source for Generating lexical-selection rules from monolingual corpora

{{TOCD}}
This page describes how to generate lexical selection rules without relying on a parallel corpus.

==Prerequisites==

* [[apertium-lex-tools]]
* [[IRSTLM]]
* A language pair (e.g. apertium-br-fr)
** The language pair should have the following two modes:
*** <code>-multi</code> which is all the modules after lexical transfer (see apertium-mk-en/modes.xml)
*** <code>-pretransfer</code> which is all the modules up to lexical transfer (see apertium-mk-en/modes.xml)

==Annotation==

'''Important:''' If you don't want through the whole process step by step, you can use the Makefile scripts provided in the [[#Makefiles|last section]] of this page.


We're going to do the example with EuroParl and the English to Spanish pair in Apertium.

Given that you've got all the stuff installed, the work will be as follows:

Make sure that you have trained a language model for the target language. See [http://sourceforge.net/apps/mediawiki/irstlm/index.php?title=User_Manual#Training_your_first_LM]

Take your corpus and make a tagged version of it:

<pre>
cat europarl.es-en.es | apertium-destxt | apertium -f none -d ~/source/apertium-en-es en-es-pretransfer > europarl.en-es.es.tagged
</pre>

Make an ambiguous version of your corpus and trim redundant tags:

<pre>
cat europarl.en-es.es.tagged | python ~/source/apertium-lex-tools/multitrans ~/source/apertium-en-es/en-es.autobil -b -f -t -n > europarl.en-es.es.ambig
</pre>

Next, generate all the possible disambiguation paths while trimming redundant tags:

<pre>
cat europarl.en-es.es.tagged | ~/source/apertium-lex-tools/multitrans ~/source/apertium-en-es/en-es.autobil -m -f -t -n > europarl.en-es.es.multi-trimmed
</pre>

Translate and score all possible disambiguation paths:

<pre>
cat europarl.en-es.es.tagged | python ~/source/apertium-lex-tools/multitrans ~/source/apertium-en-es/en-es.autobil -m -f -n |
apertium -f none -d ~/source/apertium-en-es en-es-multi | ~/source/apertium-lex-tools/irstlm-ranker 
~/source/corpora/lm/en.blm europarl.en-es.es.multi-trimmed -f > europarl.en-es.es.annotated
</pre>

Now we have a pseudo-parallel corpus where each possible translation is scored.
We start by extracting a frequency lexicon:

<pre>
	python3 ~/source/apertium-lex-tools-scripts/biltrans-extract-frac-freq.py  europarl.en-es.es.ambig europarl.en-es.es.annotated > europarl.en-es.freq
</pre>

<pre>
	python3 ~/source/apertium-lex-tools-scripts/extract-alig-lrx.py  europarl.en-es.freq > europarl.en-es.freq.lrx
</pre>

<pre>
	lrx-comp europarl.en-es.freq.lrx europarl.en-es.freq.lrx.bin
</pre>

From here on, we have two paths we can choose. We can extract rules using a maximum entropy classifier, or we can extract
rules based only on the scores provided by irstlm-ranker.

== Direct rule extraction ==
When using this method, we directly continue with extracting ngrams from the pseudo parallel corpus:

<pre>
python3 ~/source/apertium-lex-tools/scripts/biltrans-count-patterns-ngrams.py europarl.en-es.freq europarl.en-es.es.ambig europarl.en-es.es.annotated > ngrams
</pre>

Next, we prune the generated ngrams:

<pre>
python3 ~/source/apertium-lex-tools/scripts/ngram-pruning-frac.py europarl.en-es.freq ngrams > patterns
</pre>

Finally, we generate and compile lexical selection rules while thresholding their irstlm-score

<pre>
crisphold=1;
python3 ~/source/apertium-lex-tools/scripts//ngrams-to-rules.py patterns $crisphold > patterns.lrx
</pre>

<pre>
lrx-comp patterns.lrx patterns.lrx.bin
</pre>

== Maximum entropy rule extraction ==
When extracting rules using a maximum entropy criterion, we first extract features which we are going to feed to a classifier:

<pre>
python3 ~/source/apertium-lex-tools/scripts/biltrans-count-patterns-frac-maxent.py europarl-en-es.freq 
europarl.en-es.ambig europarl.en-es.annotated > events 2>ngrams
</pre>

We then train classifiers which as a side effect score how much each ngram contributes to a certain translation:
<pre>
cat events | grep -v -e '\$$ 0\.0 #' -e '\$$ 0 #' > events.trimmed
cat events.trimmed | python ~/source/apertium-lex-tools/scripts/merge-all-lambdas.py $(YASMET) > all-lambdas
</pre>

<pre>
python3 ~/source/apertium-lex-tools/scripts/merge-ngrams-lambdas.py ngrams all-lambdas > rules-all
</pre>

Finally, we extract ngrams:

<pre>
python3 ~/source/apertium-lex-tools/scripts/lambdas-to-rules.py europarl-en-es.freq rules-all > ngrams-all
</pre>

we trim them:

<pre>
python3 ~/source/apertium-lex-tools/scripts/ngram-pareto-trim.py ngrams-all > ngrams-trimmed
</pre>

and generate lexical selection rules:

<pre>
python3 ~/source/apertium-lex-tools/scripts/ngrams-to-rules-me.py ngrams-trimmed > europarl.en-es.lrx.bin
</pre>


== Makefiles ==

=== Direct rule extraction ===
You can use this makefile to generate rules using direct rule extraction. 
Your corpus needs to be placed in the same folder with your makefile.

<pre>
CORPUS=setimes
PAIR=mk-en
DATA=/home/philip/Apertium/apertium-mk-en
SL=mk
TL=en
TRAINING_LINES=10000
SCRIPTS=/home/philip/Apertium/apertium-lex-tools/scripts
MODEL=/home/philip/Apertium/corpora/language-models/en/setimes.en.5.blm
LEX_TOOLS=/home/philip/Apertium/apertium-lex-tools



AUTOBIL=$(SL)-$(TL).autobil.bin
DIR=$(SL)-$(TL)

all: data/$(CORPUS).$(DIR).freq.lrx.bin data/$(CORPUS).$(DIR).patterns.lrx

data/$(CORPUS).$(DIR).tagger: $(CORPUS).$(PAIR).$(SL)
	if [ ! -d data ]; then mkdir data; fi
	cat $(CORPUS).$(PAIR).$(SL) | head -n $(TRAINING_LINES)| sed 's/[^\.]$$/./g' | apertium-destxt | apertium -f none -d $(DATA) $(DIR)-tagger | apertium-pretransfer > $@
 
data/$(CORPUS).$(DIR).ambig: data/$(CORPUS).$(DIR).tagger
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/process-tagger-output $(DATA)/$(AUTOBIL) -b -t -f -n > $@

data/$(CORPUS).$(DIR).multi-trimmed: data/$(CORPUS).$(DIR).tagger
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/process-tagger-output $(DATA)/$(AUTOBIL) -m -t -f > $@

data/$(CORPUS).$(DIR).annotated: data/$(CORPUS).$(DIR).tagger data/$(CORPUS).$(DIR).multi-trimmed
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/process-tagger-output $(DATA)/$(AUTOBIL) -m -f | apertium -f none -d $(DATA) $(DIR)-multi | $(LEX_TOOLS)/irstlm-ranker $(MODEL) data/$(CORPUS).$(DIR).multi-trimmed -f 2>/dev/null | grep "|@|" | cut -f 1-3 > $@ 

data/$(CORPUS).$(DIR).freq: data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated
	python3 $(SCRIPTS)/biltrans-extract-frac-freq.py data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated > $@
 
data/$(CORPUS).$(DIR).freq.lrx:  data/$(CORPUS).$(DIR).freq
	python3 $(SCRIPTS)/extract-alig-lrx.py $< > $@

data/$(CORPUS).$(DIR).freq.lrx.bin: data/$(CORPUS).$(DIR).freq.lrx
	lrx-comp $< $@

data/$(CORPUS).$(DIR).ngrams: data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated
	python3 $(SCRIPTS)/biltrans-count-patterns-ngrams.py data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated > $@
 
data/$(CORPUS).$(DIR).patterns: data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ngrams
	python3 $(SCRIPTS)/ngram-pruning-frac.py data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).ngrams > $@  
 
data/$(CORPUS).$(DIR).patterns.lrx:  data/$(CORPUS).$(DIR).patterns
	python3 $(SCRIPTS)/ngrams-to-rules.py $< $(THR) > $@


</pre>

The corpus file needs to be named as "basename"."language-pair"."source side". <br/>
As an illustration, in the Makefile example, the corpus file is named europarl.en-es.es <br/>

Set the Makefile variables as follows: <br/>
CORPUS denotes the base name of your corpus file <br/>
PAIR stands for the language pair <br/>
SL and TL stand for source language and target language <br/>
DATA is the path to the language resources for the language pair <br/>
SCRIPTS denotes the path to the lex-tools scripts <br/>
MODEL is the path to the target side (binary) language model used for scoring the possible translations of ambiguous words <br/>

Finally, executing the Makefile will generate lexical selection rules for the specified language pair.

=== Maximum entropy ===
You can use this makefile to generate rules using maximum entropy classifiers. 
Your corpus needs to be placed in the same folder with your makefile.

<pre>
CORPUS=europarl
PAIR=en-es
DATA=/home/philip/source/apertium-en-es
SL=es
TL=en
MODEL=/home/philip/lm/en.blm
SCRIPTS=/home/philip/source/apertium-lex-tools/scripts
LEX_TOOLS=/home/philip/source/apertium-lex-tools
THR=1
TRAINING_LINES=10000


AUTOBIL=$(SL)-$(TL).autobil.bin
DIR=$(SL)-$(TL)
YASMET=$(LEX_TOOLS)/yasmet
all: data/$(CORPUS).$(DIR).freq.lrx.bin data/$(CORPUS).$(DIR).lm.xml

data/$(CORPUS).$(DIR).tagger: $(CORPUS).$(PAIR).$(SL)
	if [ ! -d data ]; then mkdir data; fi
	cat $(CORPUS).$(PAIR).$(SL) | head -n $(TRAINING_LINES) | apertium-destxt | apertium -f none -d $(DATA) $(DIR)-tagger | apertium-pretransfer > $@
 
data/$(CORPUS).$(DIR).ambig: data/$(CORPUS).$(DIR).tagger
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/multitrans $(DATA)/$(AUTOBIL) -b -t -f -n > $@

data/$(CORPUS).$(DIR).multi-trimmed: data/$(CORPUS).$(DIR).tagger
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/multitrans $(DATA)/$(AUTOBIL) -m -t -f > $@

data/$(CORPUS).$(DIR).annotated: data/$(CORPUS).$(DIR).tagger data/$(CORPUS).$(DIR).multi-trimmed
	cat data/$(CORPUS).$(DIR).tagger | $(LEX_TOOLS)/multitrans $(DATA)/$(AUTOBIL) -m -f | apertium -f none -d $(DATA) $(DIR)-multi | $(LEX_TOOLS)/irstlm-ranker $(MODEL) data/$(CORPUS).$(DIR).multi-trimmed -f 2>/dev/null > $@ 

data/$(CORPUS).$(DIR).freq: data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated
	python3 $(SCRIPTS)/biltrans-extract-frac-freq.py data/$(CORPUS).$(DIR).ambig data/$(CORPUS).$(DIR).annotated > $@
 
data/$(CORPUS).$(DIR).freq.lrx:  data/$(CORPUS).$(DIR).freq
	python3 $(SCRIPTS)/extract-alig-lrx.py $< > $@

data/$(CORPUS).$(DIR).freq.lrx.bin: data/$(CORPUS).$(DIR).freq.lrx
	lrx-comp $< $@

data/$(CORPUS).$(DIR).events data/$(CORPUS).$(DIR).ngrams: data/$(CORPUS).$(DIR).annotated data/$(CORPUS).$(DIR).freq
	python3 $(SCRIPTS)/biltrans-count-patterns-frac-maxent.py data/$(CORPUS).$(SL)-$(TL).freq data/$(CORPUS).$(SL)-$(TL).ambig data/$(CORPUS).$(SL)-$(TL).annotated > data/$(CORPUS).$(DIR).events 2>data/$(CORPUS).$(DIR).ngrams

data/$(CORPUS).$(DIR).all-lambdas: data/$(CORPUS).$(DIR).events
	cat data/$(CORPUS).$(DIR).events | grep -v -e '\$$ 0\.0 #' -e '\$$ 0 #' > data/$(CORPUS).$(DIR).events.trimmed
	cat data/$(CORPUS).$(DIR).events.trimmed | python $(SCRIPTS)/merge-all-lambdas.py $(YASMET)> $@

data/$(CORPUS).$(DIR).rules-all: data/$(CORPUS).$(DIR).ngrams data/$(CORPUS).$(DIR).all-lambdas
	python3 $(SCRIPTS)/merge-ngrams-lambdas.py data/$(CORPUS).$(DIR).ngrams data/$(CORPUS).$(DIR).all-lambdas > $@

data/$(CORPUS).$(DIR).ngrams-all: data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).rules-all
	python3 $(SCRIPTS)/lambdas-to-rules.py data/$(CORPUS).$(DIR).freq data/$(CORPUS).$(DIR).rules-all > $@

data/$(CORPUS).$(DIR).ngrams-trimmed: data/$(CORPUS).$(DIR).ngrams-all
	cat $< | python3 $(SCRIPTS)/ngram-pareto-trim.py > $@

data/$(CORPUS).$(DIR).lm.xml: data/$(CORPUS).$(DIR).ngrams-trimmed
	python3 $(SCRIPTS)/ngrams-to-rules-me.py data/$(CORPUS).$(DIR).ngrams-trimmed > $@

data/$(CORPUS).$(DIR).lm.bin: data/$(CORPUS).$(DIR).lm.xml
	lrx-comp $< $@


</pre>
The corpus file needs to be named as "basename"."language-pair"."source side". <br/>
As an illustration, in the Makefile example, the corpus file is named europarl.en-es.es <br/>

Set the Makefile variables as follows: <br/>
CORPUS denotes the base name of your corpus file <br/>
PAIR stands for the language pair <br/>
SL and TL stand for source language and target language <br/>
DATA is the path to the language resources for the language pair <br/>
SCRIPTS denotes the path to the lex-tools scripts <br/>
MODEL is the path to the target side (binary) language model used for scoring the possible translations of ambiguous words <br/>

Finally, executing the Makefile will generate lexical selection rules for the specified language pair.



[[Category:Lexical selection]]
[[Category:Documentation in English]]