Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Perceptron tagger

From Apertium
(Difference between revisions)
Jump to: navigation, search
Line 10: Line 10:
 
# '''Train the tagger''' like so: <code>apertium-tagger [--skip-on-error] -xs [ITERATIONS] MODEL_FILE TAGGED_CORPUS UNTAGGED_CORPUS MTX_FILE</code> which will write the resulting model to MODEL_FILE. You can put this in a Makefile. Use --skip-on-error to discard sentences for which the TAGGED and UNTAGGED corpus don't match (this can often happen as a result of the tagged corpus getting out of sync with the morphology). A reasonable value for ITERATIONS is 10.
 
# '''Train the tagger''' like so: <code>apertium-tagger [--skip-on-error] -xs [ITERATIONS] MODEL_FILE TAGGED_CORPUS UNTAGGED_CORPUS MTX_FILE</code> which will write the resulting model to MODEL_FILE. You can put this in a Makefile. Use --skip-on-error to discard sentences for which the TAGGED and UNTAGGED corpus don't match (this can often happen as a result of the tagged corpus getting out of sync with the morphology). A reasonable value for ITERATIONS is 10.
 
# '''Run the tagger''' like so: <code>apertium-tagger --tagger --perceptron MODEL_FILE</code>. You can put this in your modes.xml.
 
# '''Run the tagger''' like so: <code>apertium-tagger --tagger --perceptron MODEL_FILE</code>. You can put this in your modes.xml.
  +
  +
== Getting more information ==
  +
  +
Getting detailed information about the operation of the tagger is useful both for debugging the tagger itself as well as for designing new feature templates.
  +
  +
{| class="wikitable"
  +
! Tool
  +
! Description
  +
|-
  +
| apertium-tagger --tagger --debug
  +
| Traces the tagging process.
  +
|-
  +
| apertium-perceptron-trace model MODEL_FILE
  +
| Output the data from MODEL_FILE including the feature bytecode/disassembly and the model weights.
  +
|-
  +
| apertium-perceptron-trace path MTX_FILE UNTAGGED_CORPUS TAGGED_CORPUS
  +
| Generates features for every possible wordoid as if tagging were taking place and outputting features from TAGGED_CORPUS.
  +
|}
  +
  +
You can use apertium-tagger-trace to get the data from the model or to show which

Revision as of 18:15, 22 August 2016

The perceptron part-of-speech tagger implements part-of-speech tagging using the averaged, structured perceptron algorithm. Some information about the implementation is available in this presentation. The implementation is based on the references in the final slide.

Step by step

Mostly things are as in Supervised tagger training except you need an MTX file (and optionally a TSX file) instead of a TSX file.

    1. Get an MTX file: Copy an MTX file into your language directory and optionally modify it (or start from scratch). See MTX format.
    2. Get a tagged corpus
  1. Train the tagger like so: apertium-tagger [--skip-on-error] -xs [ITERATIONS] MODEL_FILE TAGGED_CORPUS UNTAGGED_CORPUS MTX_FILE which will write the resulting model to MODEL_FILE. You can put this in a Makefile. Use --skip-on-error to discard sentences for which the TAGGED and UNTAGGED corpus don't match (this can often happen as a result of the tagged corpus getting out of sync with the morphology). A reasonable value for ITERATIONS is 10.
  2. Run the tagger like so: apertium-tagger --tagger --perceptron MODEL_FILE. You can put this in your modes.xml.

Getting more information

Getting detailed information about the operation of the tagger is useful both for debugging the tagger itself as well as for designing new feature templates.

Tool Description
apertium-tagger --tagger --debug Traces the tagging process.
apertium-perceptron-trace model MODEL_FILE Output the data from MODEL_FILE including the feature bytecode/disassembly and the model weights.
apertium-perceptron-trace path MTX_FILE UNTAGGED_CORPUS TAGGED_CORPUS Generates features for every possible wordoid as if tagging were taking place and outputting features from TAGGED_CORPUS.

You can use apertium-tagger-trace to get the data from the model or to show which

Personal tools