Difference between revisions of "Preparing data for Moses factored training using Apertium"

From Apertium
Jump to navigation Jump to search
Line 1: Line 1:
  +
{{TOCD}}
 
This page gives a description of how to preprocess a corpus using Apertium so it can be used to train Moses factoredly.
 
This page gives a description of how to preprocess a corpus using Apertium so it can be used to train Moses factoredly.
   

Revision as of 12:26, 21 January 2010

This page gives a description of how to preprocess a corpus using Apertium so it can be used to train Moses factoredly.

Steps

Download parallel corpus

For example Europarl:

$ wget http://www.statmt.org/europarl/v5/da-en.tgz

Clean and tag both sides of corpus


Convert to Moses factored format

Download the tagger to factored script:

$ wget http://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium-tools/tagger-to-factored.py