User:Firespeaker/Cleaning up a tail - Revision history

Firespeaker at 18:56, 12 March 2014

2014-03-12T18:56:44Z

Firespeaker at 18:55, 12 March 2014

2014-03-12T18:55:40Z

Firespeaker at 18:54, 12 March 2014

2014-03-12T18:54:23Z

Firespeaker: Created page with "== The problem == For Turkic languages, there is a huge tail of unknown words when running coverage. Presumably this is because of morphological complexity—i.e., a small ha..."

2014-03-11T16:44:03Z

Created page with "== The problem == For Turkic languages, there is a huge tail of unknown words when running coverage. Presumably this is because of morphological complexity—i.e., a small ha..."

New page

== The problem ==
For Turkic languages, there is a huge tail of unknown words when running coverage. Presumably this is because of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.

== A proposed solution ==

@@ Line 1: / Line 1: @@
 == The problem ==
-[[File:Religion.unk.png|200px|Zipf's law seen in the unknown words from two Turkic corpora]]
+[[File:Religion.unk.png|thumb|Zipf's law seen in the unknown words from two Turkic corpora]]
 Due to [http://en.wikipedia.org/wiki/Zipf's%20law Zipf's law], there's a huge tail of unknown words when running coverage.  This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.

@@ Line 1: / Line 1: @@
 == The problem ==
-[[File:Religion.unk.png|thumb=200px|Zipf's law seen in the unknown words from two Turkic corpora]]
+[[File:Religion.unk.png|200px|Zipf's law seen in the unknown words from two Turkic corpora]]
 Due to [http://en.wikipedia.org/wiki/Zipf's%20law Zipf's law], there's a huge tail of unknown words when running coverage.  This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.

@@ Line 1: / Line 1: @@
 == The problem ==
+[[File:Religion.unk.png|thumb=200px|Zipf's law seen in the unknown words from two Turkic corpora]]
-For Turkic languages, there is a huge tail of unknown words when running coverage.  Presumably this is because of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.
+Due to [http://en.wikipedia.org/wiki/Zipf's%20law Zipf's law], there's a huge tail of unknown words when running coverage.  This effect is compounded in languages with high levels of morphological complexity—i.e., a small handful of unknown stems can result in hundreds of unknown forms.
+If some of these stems could be interpolated from all of their forms, transducer coverage could be increased much more quickly.
 == A proposed solution ==
+# Convert transducer to use a wildcard for a certain lemma category (especially nouns and verbs)
+# Run coverage on unknown words list
+# The top of the hitparade should include the most common unknown stems
+# Verify before adding to dictionary