Improved corpus-based paradigm matching

<spectie> we want to increase the coverage of our analysers
<spectie> one way we can do it is for a human to think about every entry
<spectie> but, we already have a lot of knowledge already
<spectie> so, if we have a load of paradigms, and a load of lemmas -- we can try and match the lemmas to paradigms by generating 
          the forms for each paradigm and looking in a corpus
<spectie> there are quite a few papers on this. 
<spectie> so, e.g. in english let's say we have the paradigms "-s" plural and "-ren" plural
<spectie> and a load of lemmas 'child, cat, dog, ...'
<spectie> we look up {child, childs}, {child, children}, {cat, catren}, {cat, cats} etc. in the corpus
<spectie> but then there is the problem of ambiguity
<spectie> in english we have ambiguity between verb forms with -s and noun forms 
<spectie> which usually isn't a big problem
<spectie> so the idea is to preprocess your corpus, using your existing analyser, and tagger to give possible values to unknown words
          based on the surrounding context
<spectie> so let's say you have: 
<spectie> na poziv *fizikalne i *matematične fakulteti u *Odesi
<spectie> you would assign possible values for case/number/gender to the unknown surface forms of *fizikalne and *matematične based 
          on the surrounding _known_ context (na .... fakulteti)
<spectie> then, when you come to check the paradigms against the surface forms in the corpus
<spectie> you're not only checking if the surface form is valid, but also if the morphological information that it predicts is valid
<spectie> as far as i know, this is novel (apart from forsberg's preliminary work)
<spectie> if you need to do some kind of machine learning, you could quite easily use the existing entries in your morphology for learning 
          which features best descriminate between ambiguous paradigms.

Example

We want to find English nouns.

We have the following paradigms:

• cat, cats
• child, children
• fox, foxes
• wolf, wolves

We can turn these into stem, suffix patterns:

• cat: - / s
• child: - / ren
• fox: - / es
• wol: f / ves

We can then extract possible candidates from our corpus, e.g.

$ cat /tmp/cand | rev | sort | rev | grep "\(f\|ves\) *$" | sort 
 aardwolf 
 aardwolves 
 behalf 
 behaves 
 belief 
 believes 
 half 
 halves 
 leaf 
 leaves 
 lives 
...

Some of these are good matches for the paradigm and others are not.

See also

• Building_dictionaries#Generating monolingual dictionary entries
• Extract