Unsupervised tagger training

First, make a directory called <lang>-tagger-data. Put the corpus you downloaded into there with a name like <lang>.crp.txt. Make sure the corpus is in raw text format with one sentence per line.

Once you have your corpus in there you need a Makefile that specifies how to generate the probability file. You can grab one from another language package. For apertium-en-af I took the Makefile from apertium-en-ca. The file that you need is called en-ca-unsupervised.make.

Copy it into your main language pair directory under an appropriate name, then edit it and change the variables at the top of the file, BASENAME, LANG1, and LANG2. Everything else should be fine.

Now run:

$ make -f en-af-unsupervised.make

and wait... you should get some output like:

Generating en-tagger-data/en.dic
This may take some time. Please, take a cup of coffee and come back later.
apertium-validate-dictionary apertium-en-af.en.dix
apertium-validate-tagger apertium-en-af.en.tsx
lt-expand apertium-en-af.en.dix | grep -v "__REGEXP__" | grep -v ":<:" |\
        awk 'BEGIN{FS=":>:|:"}{print $1 ".";}' | apertium-destxt >en.dic.expanded
lt-proc -a en-af.automorf.bin <en.dic.expanded | \
        apertium-filter-ambiguity apertium-en-af.en.tsx > en-tagger-data/en.dic
rm en.dic.expanded;
apertium-destxt < en-tagger-data/en.crp.txt | lt-proc en-af.automorf.bin > en-tagger-data/en.crp
apertium-validate-tagger apertium-en-af.en.tsx
apertium-tagger -t 8 \
                           en-tagger-data/en.dic \
                           en-tagger-data/en.crp \
                           apertium-en-af.en.tsx \
                           en-af.prob;
Calculating ambiguity classes...
Kupiec's initialization of transition and emission probabilities...
Applying forbid and enforce rules...
Training (Baum-Welch)...
Applying forbid and enforce rules...

And after this you should have a en-af.prob file, which can be used with the apertium-tagger module.

Some questions and answers about unsupervised tagger training

Q: How big a dictionary do I need?

A: For English and Esperanto we had approx 13000 entries. Approx half of the training sentences had an unknown word. With this we got very poor tagger performance. Then we added 7000 proper nouns, so we had 20000 entries. That made the quality acceptable.

Q: My dix is not big enough, and approx half of the training sentences has an unknown word. Can't I just grep these sentences away, and then train on the rest?

A: No. Unknown words gets a special category, so you also needs some adequate representation of unknown words in your training set.

Q: In which circonstances can I just copy a tagger .prob file (or a .tsx file) from another project?

A: You must make sure that the symbols are exactly the same. For example eo-en uses symbols have<vblex><pres><p3><sg> and es-en uses have<pri><pres><p3><sg>, so they will not work.

Q: I changed a paradigm which is often used and now a lot of the words that uses that paradigm are tagged differently!

A: Yes. You will need to retrain your tagger becaurse the probablilites have changed. If you for example remove imperative (which in English is the same as the infinitive) for a verb paradigm the tagger will distribute the probablities to the other possibilites.

Q: Can I make the tagger distinguish between surface forms that are the same in all circonstances.

A: Probably not very well. For example in English imperative has the same form as the infinitive. Unless you write some extremely clever TSX rules the tagger has no change of distinguishing the two forms and will select between them more or less randomly. Such things are much better detected and handled in transfer.

Improving the tagger performance

Q: My tagger is performin poorly. What can I do?

A: Assuming that your TSX file is OK, the best thing you can do is to add words to your dix so less words (but still some) are unknown.

Q: Can't I just tag a corpus with the tagger, correct the tags in places where it has selected the wrong possibility, and retrain on that file?

A: ??????

Q: Can I improve my unsupervised training with selected by-hand disambiguated examples?

A: You can train with a new iteration taking the probabilities from the previous training with the option --retrain. Categories must be the same, the tsx file must be the same. The expert here is felipeç. He said:

The option --retrain is used to retrain the tagger: In each iteration of Baum Welch, the probabilities of the Markov model are reestimated using the probabilities obtained in the previous iteration. With --retrain what you are saying to the tagger is to read the probabilities of a file and reestimate them with the training corpus; in other words, to add one or more iterations. For example, training with 6 iterations and retraining with 2 is equivalent to training with 8 iterations from the beginning (supposing that it has the same corpus, of course).

A way to mix supervised and unsupervised training is to train supervisedly with a manually labelled (disambiguated) corpus and afterwards re-train (--retrain) with a bigger unlabelled corpus.