Supervised tagger training
Supervised tagger training is the manual way of training the tagger. It takes more time than unsupervised tagger training, but is also more effective.
What you need
It is named like this:
lang1-lang2-supervised.make, for example,
en-eo-supervised.make. If you don't already have one in the language pair directory, you can copy this one from en-eo. You will need modify it to fit your language pair. This usually means editing the first few lines.
Tagger data directory
This is a directory called
lang1-tagger-data, for example,
en-tagger-data in case of English. This directory should be inside the language pair directory. Create one if it is not there.
This is the source text which you will use for training. For supervised training, it is recommended to have a large corpus of over 30,000 words. You can find some here. Enter the directory corresponding the language you want to train with, and look for files ending with
.raw.txt. Those won't usually be more than 2000 words each, but if you combine all raw files into a single file, you should get a corpus large enough. Do this, and save that one file as
lang1.tagged.txt inside the directory
A handtagged copy of that corpus
This repo also contains the handtagged versions of each raw file. These end with
.handtagged.txt. In some cases, there may be more than one handtagged versions of each file. You can check the difference between these and choose the most correct one. Combine these chosen handtagged files into one file, in the same order as you did with raw files. Save this as
Inside your language pair directory, run:
make -f lang1-lang2-supervised.make
lang2 with the corresponding language codes.
If everything is set up correctly, this will generate a new file called
lang1-tagger-data. This file is the tagger's interpretation of your corpus -- the machine-tagged file (although it is named
Normally, the command above will end up with an error. It is fine. This happens because the words in
lang1.untagged do not match each other. For example, a group of words could be a multiword according to the handtagged file, but not according to the machine-tagged file, or vice versa. This error will be solved only when the tagset in
lang1.tagged is equivalent to one of the possible tagsets mentioned in
There are two common ways of solving this:
- Edit the
lang1.taggedfile, so that it matches the tags expected by
- Edit the dix, so that the tagger understands the new wordform and tags correctly the next time.
Which methods to choose, depends on the particular words in question, and it is up to you to decide. In any case, do not edit the
lang1.untagged file. It is autogenerated, and all your changes will be lost anyway.
Once the mismatch is solved, run the above command again to check if it worked. If you get a different error from last time, it worked. Now keep solving all the mismatches until there are no more errors. When the execution of the command finishes without errors, the training is complete.