Difference between revisions of "Building a pseudo-parallel corpus"
Line 17: | Line 17: | ||
</pre> |
</pre> |
||
optimization |
Revision as of 10:50, 23 August 2012
Acquiring parallel corpora can be a difficult process and for some language pairs such resources might not exist. However, we can use a language model for the target language in order to create pseudo-parallel corpora, and use them in the same way as parallel ones.
IRSTLM
IRSTLM is a tool for building n-gram language models from corpora. It supports different smoothing and interpolation methods, including Written-Bell smoothing, Kneser-Ney smoothing and others.
The full documentation can be viewed here, and the whole toolkit can be downloaded here
Building a pseudo-parallel corpus
The main idea is to get a source-language corpus and run it through the apertium pipeline, but this time let the language model choose the preposition instead of apertium. The main algorithm is as follows (example for mk-en):
run the corpus through mk-en-biltrans Run through <code>apertium-lex-tools/scripts/biltrans-to-multitrans.py</code> Run through the rest of the pipeline from apertium-transfer -b onwards Run through <code>apertium-lex-learner/irstlm-ranker<code>