Supervised tagger training

Supervised tagger training is the manual way of training the tagger. It takes more time than unsupervised tagger training, but is also more effective.

What you need

A makefile

It is named like this: lang1-lang2-supervised.make, for example, en-eo-supervised.make. If you don't already have one in the language pair directory, you can copy this one from en-eo. You will need modify it to fit your language pair. This usually means editing the first few lines.

Tagger data directory

This is a directory called lang1-tagger-data, for example, en-tagger-data in case of English. This directory should be inside the language pair directory. Create one if it is not there.

A handtagged corpus

This repo also contains many handtagged files for every language. These end with .handtagged.txt. In some cases, there may be more than one handtagged versions of each file. You can check the difference between these and choose the most correct one. Combine these chosen handtagged files into one file, and save it as lang1.tagged inside lang1-tagger-data.

Procedure

First, you need to extract raw text from your handtagged files. Enter lang1-tagger-data and run:

cat lang1.tagged | cut -f2 -d'^' | cut -f1 -d'/' > lang1.tagged.txt

Replace lang1 with the corresponding language code.

Next, cd .. up into your language pair directory and run:

make -f lang1-lang2-supervised.make

If everything is set up correctly, this will generate a new file called lang1.untagged inside lang1-tagger-data. This file is the tagger's interpretation of your corpus -- the machine-tagged file (although it is named untagged).

Normally, the command above will end up with an error. It is fine. This happens because the words in lang1.tagged and lang1.untagged do not match each other. For example, a group of words could be a multiword according to the handtagged file, but not according to the machine-tagged file, or vice versa. This error will be solved only when the tagset in lang1.tagged is equivalent to one of the possible tagsets mentioned in lang1.untagged.

There are two common ways of solving this:

• Edit the lang1.tagged file, so that it matches the tags expected by lang1.untagged.
• Edit the dix, so that the tagger understands the new wordform and tags correctly the next time.

Which methods to choose, depends on the particular words in question, and it is up to you to decide. In any case, do not edit the lang1.untagged file. It is autogenerated, and all your changes will be lost anyway.

Once the mismatch is solved, run the above command again to check if it worked. If you get a different error from last time, it worked. Now keep solving all the mismatches until there are no more errors. When the execution of the command finishes without errors, the training is complete.