Supervised tagger training

Supervised tagger training is the manual way of training the tagger. It takes more time than unsupervised tagger training, but is also more effective.

What you need

A makefile

It is named like this: lang1-lang2-supervised.make, for example, en-eo-supervised.make. If you don't already have one in the language pair directory, you can copy this one from en-eo. You will need modify it to fit your language pair. This usually means editing the first few lines.

Tagger data directory

This is a directory called lang1-tagger-data, for example, en-tagger-data in case of English. This directory should be inside the language pair directory. Create one if it is not there.

A corpus

This is the source text which you will use for training. For supervised training, it is recommended to have a large corpus of over 30,000 words. You can find some here. Enter the directory corresponding the language you want to train with, and look for files ending with .raw.txt. Those won't usually be more than 2000 words each, but if you combine all raw files into a single file, you should get a corpus large enough. Do this, and save that one file as lang1.tagged.txt inside the directory lang1-tagger-data.

A handtagged copy of that corpus

This repo also contains the handtagged versions of each raw file. These end with .handtagged.txt. In some cases, there may be more than one handtagged versions of each file. You can check the difference between these and choose the most correct one. Combine these chosen handtagged files into one file, in the same order as you did with raw files. Save this as lang1.tagged inside lang1-tagger-data.

Procedure

Inside your language pair directory, run:

make -f lang1-lang2-supervised.make

Replace lang1 and lang2 with the corresponding language codes.

If everything is set up correctly, this will generate a new file called lang1.untagged inside lang1-tagger-data. This file is the tagger's interpretation of your corpus -- the machine-tagged file (although it is named untagged).

Normally, the command above will end up with an error. It is fine. This happens because the words in lang1.tagged and lang1.untagged do not match each other. For example, a group of words could be a multiword according to the handtagged file, but not according to the machine-tagged file, or vice versa. This error will be solved only when the tagset in lang1.tagged is equivalent to one of the possible tagsets mentioned in lang1.untagged.

There are two common ways of solving this:

• Edit the lang1.tagged file, so that it matches the tags expected by lang1.untagged.
• Edit the dix, so that the tagger understands the new wordform and tags correctly the next time.

Which methods to choose, depends on the particular words in question, and it is up to you to decide. In any case, do not edit the lang1.untagged file. It is autogenerated, and all your changes will be lost anyway.

Once the mismatch is solved, run the above command again to check if it worked. If you get a different error from last time, it worked. Now keep solving all the mismatches until there are no more errors. When the execution of the command finishes without errors, the training is complete.