Improved corpus-based paradigm matching - Revision history

Francis Tyers: /* See also */

2015-02-10T14:26:26Z

Francis Tyers: /* Example */

2015-02-10T13:50:19Z

Example

Francis Tyers: /* Example */

2015-02-10T13:44:56Z

Example

Francis Tyers at 13:41, 10 February 2015

2015-02-10T13:41:05Z

Unhammer at 13:14, 10 February 2015

2015-02-10T13:14:24Z

Unhammer at 13:11, 10 February 2015

2015-02-10T13:11:14Z

Francis Tyers at 21:39, 1 October 2012

2012-10-01T21:39:12Z

Francis Tyers: Created page with '

  we want to increase the coverage of our analysers  one way we can do it is for a human to think about every entry  but, we already have a lot o…'

2012-10-01T21:32:58Z

Created page with ' <pre> <spectie> we want to increase the coverage of our analysers <spectie> one way we can do it is for a human to think about every entry <spectie> but, we already have a lot o…'

New page

<pre>
<spectie> we want to increase the coverage of our analysers
<spectie> one way we can do it is for a human to think about every entry
<spectie> but, we already have a lot of knowledge already
<spectie> so, if we have a load of paradigms, and a load of lemmas -- we can try and match the lemmas to paradigms by generating
the forms for each paradigm and looking in a corpus
<spectie> there are quite a few papers on this.
<spectie> so, e.g. in english let's say we have the paradigms "-s" plural and "-ren" plural
<spectie> and a load of lemmas 'child, cat, dog, ...'
<spectie> we look up {child, childs}, {child, children}, {cat, catren}, {cat, cats} etc. in the corpus
<spectie> but then there is the problem of ambiguity
<spectie> in english we have ambiguity between verb forms with -s and noun forms
<spectie> which usually isn't a big problem
<spectie> so the idea is to preprocess your corpus, using your existing analyser, and tagger to give possible values to unknown words
based on the surrounding context
<spectie> so let's say you have:
<spectie> na poziv *fizikalne i *matematične fakulteti u *Odesi
<spectie> you would assign possible values for case/number/gender to the unknown surface forms of *fizikalne and *matematične based
on the surrounding _known_ context (na .... fakulteti)
<spectie> then, when you come to check the paradigms against the surface forms in the corpus
<spectie> you're not only checking if the surface form is valid, but also if the morphological information that it predicts is valid
<spectie> as far as i know, this is novel
</pre>

[[Category:Development]]

@@ Line 99: / Line 99: @@
 * [[Building_dictionaries#Generating monolingual dictionary entries]]
 * [[Extract]]
+==Further reading==
+* http://spraakbanken.gu.se/lb/personal/markus/publications/FinTAL2006.pdf
+* http://spraakbanken.gu.se/lb/personal/markus/publications/Extract_Tech_Report.pdf
 [[Category:Development]]

@@ Line 63: / Line 63: @@
 Some of these are good matches for the paradigm and others are not.
+How can we determine if something is a good match or not ? We can use context. So our candidates are:
+* aardwolf, aardwolves
+* belief, believes
+* half, halves
+* leaf, leaves
+As these all follow the pattern +f in singular and +ves in plural. Let's say to start we want to make sure that all of them are nouns, e.g. because "believes" is a verb and we don't want that.
+We could say e.g. that we want to make sure that every part of the paradigm can be found with an article before it.
+<pre>
+$ cat /tmp/sample | grep 'the aardwolf' | wc -l
+$ cat /tmp/sample | grep 'the aardwolves' | wc -l
+$ cat /tmp/sample | grep 'the belief' | wc -l
+$ cat /tmp/sample | grep 'the believes' | wc -l
+$ cat /tmp/sample | grep 'the half' | wc -l
+$ cat /tmp/sample | grep 'the halves' | wc -l
+$ cat /tmp/sample | grep 'the leaf' | wc -l
+$ cat /tmp/sample | grep 'the leaves' | wc -l
+</pre>
+So going by this output (with a very small corpus), you would accept the first pair, and reject the others.
 ==See also==

@@ Line 44: / Line 44: @@
 * wol: f / ves
+We can then extract possible candidates from our corpus, e.g.
+<pre>
+$ cat /tmp/cand | rev | sort | rev | grep "\(f\|ves\) *$" | sort
+ aardwolf
+ aardwolves
+ behalf
+ behaves
+ belief
+ believes
+ half
+ halves
+ leaf
+ leaves
+ lives
+...
+</pre>
+Some of these are good matches for the paradigm and others are not.
 ==See also==

@@ Line 24: / Line 24: @@
           which features best descriminate between ambiguous paradigms.
 </pre>
+==Example==
+We want to find English nouns.
+We have the following paradigms:
+* cat, cats
+* child, children
+* fox, foxes
+* wolf, wolves
+We can turn these into stem, suffix patterns:
+* cat: - / s
+* child: - / ren
+* fox: - / es
+* wol: f / ves
 ==See also==

@@ Line 26: / Line 26: @@
 ==See also==
-* [[Building_dictionaries#Monodix]]
+* [[Building_dictionaries#Generating monolingual dictionary entries]]
 * [[Extract]]

@@ Line 1: / Line 1: @@
 <pre>
 <spectie> we want to increase the coverage of our analysers
@@ Line 6: / Line 5: @@
 <spectie> so, if we have a load of paradigms, and a load of lemmas -- we can try and match the lemmas to paradigms by generating
           the forms for each paradigm and looking in a corpus
 <spectie> there are quite a few papers on this.
 <spectie> so, e.g. in english let's say we have the paradigms "-s" plural and "-ren" plural
 <spectie> and a load of lemmas 'child, cat, dog, ...'
@@ Line 21: / Line 20: @@
 <spectie> then, when you come to check the paradigms against the surface forms in the corpus
 <spectie> you're not only checking if the surface form is valid, but also if the morphological information that it predicts is valid
-<spectie> as far as i know, this is novel
+<spectie> as far as i know, this is novel (apart from forsberg's preliminary work)
+<spectie> if you need to do some kind of machine learning, you could quite easily use the existing entries in your morphology for learning
+          which features best descriminate between ambiguous paradigms.
 </pre>