Generating lexical-selection rules from monolingual corpora

From Apertium
Jump to navigation Jump to search

This page describes how to generate lexical selection rules without relying on a parallel corpus.


  • apertium-lex-tools
  • A language pair (e.g. apertium-br-fr)
    • The language pair should have the following two modes:
      • -multi which is all the modules after lexical transfer (see apertium-mk-en/modes.xml)
      • -pretransfer which is all the modules up to lexical transfer (see apertium-mk-en/modes.xml)


Take your corpus and run it through the lexical transfer:

cat $(CORPUS).$(DIR).txt | apertium-destxt | apertium -f none -d $(DATA) $(DIR)-pretransfer | lt-proc -b $(DATA)/$(AUTOBIL) > $@

Then select only the lines which have more than one and less than 10,000 translations, which have an ambiguous noun/verb/adjective and which have >= 90% coverage of the morphology.

cat $< | python3 $(SCRIPTS)/ | python3 $(SCRIPTS)/ | python3 $(SCRIPTS)/ > $@

Generate all the possible disambiguation paths:

cat $< | python $(SCRIPTS)/ > $@

Translate all possible disambiguation paths:

cat $< | apertium -f none -d $(DATA) $(DIR)-multi > $@

Score all the possible disambiguation paths with IRSTLM.


First extract the default translations:

Then the ngram partial counts:

Finding the best threshold