Unigram tagger

From Apertium
Revision as of 17:52, 16 January 2016 by M5w (talk | contribs) (→‎Model 3)
Jump to navigation Jump to search

apertium-tagger from “m5w/apertium”[1] supports all the unigram models from “A set of open-source tools for Turkish natural language processing.”[2]

Installation

First, install all prerequisites. See “If you want to add language data / do more advanced stuff.”[3]

Then, replace <directory> with the directory you’d like to clone “m5w/apertium”[1] into and clone the repository.

git clone https://github.com/m5w/apertium.git <directory>

Then, configure your environment[4] and finally configure, build, and install[5] “m5w/apertium.”[1]

Usage

See apertium-tagger -h .

Training a Model on a Hand-Tagged Corpus

First, get a hand-tagged corpus as you would for any non-unigram model.

$ cat handtagged.txt
^a/a<a>$
^a/a<b>$
^a/a<b>$
^aa/a<a>+a<a>$
^aa/a<a>+a<b>$
^aa/a<a>+a<b>$
^aa/a<b>+a<a>$
^aa/a<b>+a<a>$
^aa/a<b>+a<a>$
^aa/a<b>+a<b>$
^aa/a<b>+a<b>$
^aa/a<b>+a<b>$
^aa/a<b>+a<b>$

Example 2.1.1: handtagged.txt : a Hand-Tagged Corpus for apertium-tagger

Then, replace MODEL with the unigram model from “A set of open-source tools for Turkish natural language processing”[2] you’d like to use, replace SERIALISED_BASIC_TAGGER with the filename to which you’d like to write the model, and train the tagger.

$ apertium-tagger -s 0 -u MODEL SERIALISED_BASIC_TAGGER handtagged.txt

Disambiguation

Either write your input to a file or pipe it to the tagger.

$ cat raw.txt
^a/a<a>/a<b>/a<c>$
^aa/a<a>+a<a>/a<a>+a<b>/a<b>+a<a>/a<b>+a<b>/a<a>+a<c>/a<c>+a<a>/a<c>+a<c>$

Example 2.2.1: raw.txt : Input for apertium-tagger

Replace MODEL with the unigram model from “A set of open-source tools for Turkish natural language processing”[2] you’d like to use, replace SERIALISED_BASIC_TAGGER with the file to which you wrote the unigram model, and disambiguate the input.

$ apertium-tagger -gu MODEL SERIALISED_BASIC_TAGGER raw.txt
^a/a<b>$
^aa/a<b>+a<b>$
$ echo '^a/a<a>/a<b>/a<c>$
^aa/a<a>+a<a>/a<a>+a<b>/a<b>+a<a>/a<b>+a<b>/a<a>+a<c>/a<c>+a<a>/a<c>+a<c>$' | \
apertium-tagger -gu MODEL SERIALISED_BASIC_TAGGER
^a/a<b>$
^aa/a<b>+a<b>$

Unigram Models

See section 5.3 of “A set of open-source tools for Turkish natural language processing.”[2]

Model 1

See section 5.3.1 of “A set of open-source tools for Turkish natural language processing.”[2]

This model assigns each analysis string a score of

with additive smoothing.

Consider the following corpus.

$ cat handtagged.txt
^a/a<a>$
^a/a<b>$
^a/a<b>$

Example 3.1.1: handtagged.txt : A Hand-Tagged Corpus for apertium-tagger

Given the lexical unit ^a/a<a>/a<b>/a<c>$ , the tagger assigns the analysis string a<a> a score of

The tagger then assigns the analysis string a<b> a score of

and the unknown analysis string a<c> a score of

If ./autogen.sh is passed the option --enable-debug , the tagger prints such calculations to standard error.

$ ./autogen.sh --enable-debug
$ make
$ echo '^a/a<a>/a<b>/a<c>$' | apertium-tagger -gu 1 SERIALISED_BASIC_TAGGER


score("a<a>") ==
  2 ==
  2.000000000000000000
score("a<b>") ==
  3 ==
  3.000000000000000000
score("a<c>") ==
  1 ==
  1.000000000000000000
^a<b>$

Training on Corpora with Ambiguous Lexical Units

Consider the following corpus.

$ cat handtagged.txt
^a/a<a>$
^a/a<a>/a<b>$
^a/a<b>$
^a/a<b>$

Example 3.1.1.1: handtagged.txt : a Hand-Tagged Corpus for apertium-tagger

The tagger expects lexical units of 1 analysis string, or lexical units of size 1. However, the size of the lexical unit ^a/a<a>/a<b>$ is 2. For this lexical unit,

the tagger must effectively increment the frequency of both analysis strings by 0.500000000000000000 . However, the tagger can’t increment the analysis strings’ frequencies by a non-integral number because model 1 represents analysis strings’ frequencies as std::size_t [6].

Instead, the tagger multiplies all the stored analysis strings’ frequencies by this lexical unit’s size and increments the frequency of each of this lexical unit’s analysis strings by 1.

The tagger could then increment the analysis strings’ frequencies of another lexical unit of size 2 without multiplying any of the stored analysis strings’ frequencies. To account for this, the tagger stores the least common multiple of all lexical units’ sizes; only if the LCM isn’t divisible by a lexical unit’s size does the tagger multiply all the analysis strings’ frequencies.

After incrementing the analysis strings’ frequencies of the lexical unit ^a/a<a>/a<b>$, the tagger increments the analysis string a<b> of the lexical unit ^a/a<b>$ by

If the tagger gets another lexical unit of size 2, it would increment the frequency of each of the lexical unit’s analysis strings by

and if it gets a lexical unit of size 3, it would multiply all the analysis strings’ frequencies by 3 and then increment the frequency of each of the lexical unit’s analysis strings by

Each model supports functions to increment all their stored analysis strings’ frequencies, so models 2 and 3 support this algorithm as well.

TODO: If one passes the -d option to apertium-tagger , the tagger prints warnings about ambiguous analyses in corpora to stderr.

$ apertium-tagger -ds 0 -u 1 handtagged.txt
apertium-tagger: handtagged.txt: 2:13: unexpected analysis "a<b>" following anal
ysis "a<a>"
^a/a<a>/a<b>$
            ^

Model 2

See section 5.3.2 of “A set of open-source tools for Turkish natural language processing.”[2]

Consider Example 3.1.1: handtagged.txt .

The tag string <b> is twice as frequent as <a>. However, model 1 scores b<a> and b<b> equally because neither analysis string appears in the corpus.

This model splits each analysis string into a root, , and the part of the analysis string that isn’t the root, . An analysis string’s root is its first lemma. The of a<b>+c<d> is a ; its is <b>+c<d> . The tagger assigns each analysis string a score of with additive smoothing. (Without additive smoothing, this model would be the same as model 1.)[7] The tagger assigns higher scores to unknown analysis strings with frequent than to unknown analysis strings with infrequent .

Given the lexical unit ^b/b<a>/b<b>$, the tagger assigns the analysis string b<a> a score of

Note that counts the analysis string being scored. For example, the tagger would assign the known analysis string a<a> a score of

The tagger assigns the analysis string b<b> a score of

Model 3

This model splits each analysis string into an a first inflection, and a series of derivation-inflection pairs, The of the analysis string a<b>+c<d> is a , its is <b> , and its is

Notes

  1. 1.0 1.1 1.2 https://github.com/m5w/apertium
  2. 2.0 2.1 2.2 2.3 2.4 2.5 http://coltekin.net/cagri/papers/trmorph-tools.pdf
  3. Installation#If you want to add language data / do more advanced stuff
  4. Minimal installation from SVN#Set up environment
  5. Minimal installation from SVN#Configure, build, and install
  6. http://en.cppreference.com/w/cpp/types/size_t