Difference between revisions of "User:Firespeaker/Cleaning up a tail"
Jump to navigation
Jump to search
Firespeaker (talk | contribs) |
Firespeaker (talk | contribs) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
== The problem == |
== The problem == |
||
[[File:Religion.unk.png|thumb |
[[File:Religion.unk.png|thumb|Zipf's law seen in the unknown words from two Turkic corpora]] |
||
Due to [http://en.wikipedia.org/wiki/Zipf's%20law Zipf's law], there's a huge tail of unknown words when running coverage. This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms. |
Due to [http://en.wikipedia.org/wiki/Zipf's%20law Zipf's law], there's a huge tail of unknown words when running coverage. This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms. |
||
Latest revision as of 18:56, 12 March 2014
The problem[edit]
Due to Zipf's law, there's a huge tail of unknown words when running coverage. This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.
If some of these stems could be interpolated from all of their forms, transducer coverage could be increased much more quickly.
A proposed solution[edit]
- Convert transducer to use a wildcard for a certain lemma category (especially nouns and verbs)
- Run coverage on unknown words list
- The top of the hitparade should include the most common unknown stems
- Verify before adding to dictionary
