Supervised tagger training
Supervised tagger training is the manual way of training the tagger. It takes more time than unsupervised tagger training, but is also more effective.
What you need
It is named like this:
lang1-lang2-supervised.make, for example,
en-eo-supervised.make. If you don't already have one in the language pair directory, you can copy this one from en-eo. You will need modify it to fit your language pair. This usually means editing the first few lines.
Tagger data directory
This is a directory called
lang1-tagger-data, for example,
en-tagger-data in case of English. This directory should be inside the language pair directory. Create one if it is not there.
A handtagged corpus
This repo also contains many handtagged files for every language. These end with
.handtagged.txt. In some cases, there may be more than one handtagged versions of each file. You can check the difference between these and choose the most correct one. Combine these chosen handtagged files into one file, and save it as
First, you need to extract raw text from your handtagged files. Enter
lang1-tagger-data and run:
cat lang1.tagged | cut -f2 -d'^' | cut -f1 -d'/' > lang1.tagged.txt
lang1 with the corresponding language code.
cd .. up into your language pair directory and run:
make -f lang1-lang2-supervised.make
If everything is set up correctly, this will generate a new file called
lang1-tagger-data. This file is the tagger's interpretation of your corpus -- the machine-tagged file (although it is named
Normally, the command above will end up with an error. It is fine. This happens because the words in
lang1.untagged do not match each other. For example, a group of words could be a multiword according to the handtagged file, but not according to the machine-tagged file, or vice versa. This error will be solved only when the tagset in
lang1.tagged is equivalent to one of the possible tagsets mentioned in
There are two common ways of solving this:
- Edit the
lang1.taggedfile, so that it matches the tags expected by
- Edit the dix, so that the tagger understands the new wordform and tags correctly the next time.
Which methods to choose, depends on the particular words in question, and it is up to you to decide. In any case, do not edit the
lang1.untagged file. It is autogenerated, and all your changes will be lost anyway.
Once the mismatch is solved, run the above command again to check if it worked. If you get a different error from last time, it worked. Now keep solving all the mismatches until there are no more errors. When the execution of the command finishes without errors, the training is complete.