Finding errors in dictionaries - Revision history

Bech: wikifiying presentation

2015-03-15T13:14:26Z

wikifiying presentation

Bech: More correct title for French page

2015-03-15T12:52:06Z

More correct title for French page

Tunedal: /* Make a list of correctly spelled words */

2015-02-17T13:45:13Z

Make a list of correctly spelled words

Tunedal: /* Make a list of correctly spelled words */

2015-02-17T12:58:09Z

Make a list of correctly spelled words

Tunedal: /* Make a list of correctly spelled words */

2015-02-17T12:26:01Z

Make a list of correctly spelled words

Tunedal: /* Spell-check the rest of the expanded dictionary */

2015-02-17T08:44:20Z

Spell-check the rest of the expanded dictionary

Tunedal: /* Spell-check the rest of the expanded dictionary */

2015-02-17T08:42:01Z

Spell-check the rest of the expanded dictionary

Tunedal: /* Exclude the correctly spelled words */

2015-02-17T08:24:33Z

Exclude the correctly spelled words

Tunedal: /* Expand the monodix */

2015-02-16T18:38:37Z

Expand the monodix

Tunedal: /* Option: check for duplicate entries */

2015-02-16T15:43:09Z

Option: check for duplicate entries

@@ Line 7: / Line 7: @@
 == Summary ==
-First expand the monodix. This will produce a very large file.
+* First expand the monodix. This will produce a very large file.
-Continue  by making a list of correctly spelled words and exclude them from the expanded dictionary.
+* Continue  by making a list of correctly spelled words and exclude them from the expanded dictionary.
-Finally check the remaining words in a word processing program of your choice to quickly find the errors. Open the original dictionary file in an editor and correct the errors you have found.
+* Finally check the remaining words in a word processing program of your choice to quickly find the errors. Open the original dictionary file in an editor and correct the errors you have found.
-Option: Check for duplicate entries.
+* Option: Check for duplicate entries.
 == Expand the monodix ==
 Move to the folder where the dictionary is kept. The following command expands the dictionary, i.e. creates all forms of every word according to the assigned paradigm. Only the forms that are not marked by any LR or RL tag are expanded and the erroneous entries causes by a long known bug (NON_ANALYSIS) are filtered away. The example below expands the Swedish monolingual dictionary:
-lt-expand apertium-swe.swe.dix | grep -v ':[<>]:' | cut -f1 -d:|
+ lt-expand apertium-swe.swe.dix | grep -v ':[<>]:' | cut -f1 -d:| fgrep -v 'NON_ANALYSIS' > swe.expanded
-fgrep -v 'NON_ANALYSIS' > swe.expanded
-Change to the dictionary you would like to correct, i.e. change "apertium-swe.swe.dix" to the name of your dictionary and change the output name from "swe.expanded" to something appropriate.
+Change to the dictionary you would like to correct, i.e. change "<code>apertium-swe.swe.dix</code>" to the name of your dictionary and change the output name from "<code>swe.expanded</code>" to something appropriate.
 == Make a list of correctly spelled words ==
@@ Line 28: / Line 27: @@
 You can get a list of correct words from Aspell. The following command gets a list of Swedish words:
-aspell -d sv dump master | aspell -l sv expand > aspellwords.sv
+ aspell -d sv dump master | aspell -l sv expand > aspellwords.sv
 Just change the language code for the language you are working with. For e.g. English it would be:
-aspell -d en dump master | aspell -l en expand > aspellwords.en
+ aspell -d en dump master | aspell -l en expand > aspellwords.en
 You'll find more info in the Aspell manual if needed.
@@ Line 42: / Line 41: @@
 You can get a frequency list for instance with the following command:
+<pre>
 cat my_swedish_corpus.txt | tr ' ' '\n' |
 tr '[:upper:]' '[:lower:]' |
 tr -d '[:punct:]' | grep -v '[^a-z]' |
 sort | uniq -c | sort -rn > frequency.sv
+</pre>
 Open the frequency list in an editor of your choice. Browsing the list you'll find that the frequency drops quickly. Simply delete all words that not are common and save the rest as e.g. top_frequency.sv (or whatever appropriate). Change the command above to what suits the name of your corpus and the language you are working with.
@@ Line 54: / Line 55: @@
 When you've got a nice long list of correctly spelled words to exclude, filter the expanded wordlist from Apertium. This is easy to accomplish with grep. The following command would for instance filter the expanded Swedish monolingual dictionary:
+<pre>
-cat swe.expanded |
-grep -v -wFf  top_frequency.sv |
+cat swe.expanded | grep -v -wFf  top_frequency.sv |
 grep -v -wFf  aspellwords.sv > swe.expanded.felstavade
+</pre>
 You would find the suspected errors in the file swe.expanded.felstavade (felstavade = misspelled).
@@ Line 79: / Line 81: @@
 It might happen that there are duplicate entries for the same word. You can easily find them if you make a frequency list out of the expanded dictionary. This will make a frequency list for the Swedish expanded dictionary:
+<pre>
 lt-expand *.swe.dix | grep -v ':[<>]:' | cut -f1 -d:|
 fgrep -v 'NON_ANALYSIS' | sort |
 uniq -c | sort -rn > swe.expanded.freq
+</pre>
 Start at the top and check if there might be duplicates. Please note two common false alarms:

← Older revision		Revision as of 12:52, 15 March 2015
Line 1:		Line 1:
	[[~~Trouver_des_erreurs_dans_des_dictionaries~~\|En français]]		[[Trouver_des_erreurs_dans_des_dictionnaires\|En français]]

	{{TOCD}}		{{TOCD}}

@@ Line 49: / Line 49: @@
 Open the frequency list in an editor of your choice. Browsing the list you'll find that the frequency drops quickly. Simply delete all words that not are common and save the rest as e.g. top_frequency.sv (or whatever appropriate). Change the command above to what suits the name of your corpus and the language you are working with.
-You can read more about getting a corpus and making a frequency list at the page [[Building_dictionaries]].
+You can read more about getting a corpus and making a frequency list at the page [[Building_dictionaries|Building dictionaries]].
 == Exclude the correctly spelled words ==

@@ Line 36: / Line 36: @@
 You'll find more info in the Aspell manual if needed.
-This list is however rather short. You might find it useful to filter on more words. One way of getting more correctly spelled words is to simply use the top of a word frequency list made on a large corpus. Rational: most people spell correctly most of the time. Thus highly frequent words are most probably correctly spelled. If they are not, they will probably be the new standard for spelling :-)
+The list from Aspell is however rather short. You might find it useful to filter on more words. One way of getting more correctly spelled words is to simply use the top of a word frequency list made on a large corpus. Rational: most people spell correctly most of the time. Thus highly frequent words are most probably correctly spelled. If they are not, they will probably be the new standard for spelling :-)
 You can download a corpus from eg. OPUS [http://opus.lingfil.uu.se/ OPUS Uppsala University, Sweden]. Choose among Europarl, OpenOffice and OpenSubtitles etc in many languages.

@@ Line 24: / Line 24: @@
 == Make a list of correctly spelled words ==
-The expanded word list is a very large haystack to look for needles in. To make the task somewhat easier you would like to get rid off as much hay as possible, without throwing away any needles. An easy way is to simply drop all words that are spelled correctly. This can be done by filtering the list against a list of correctly spelled words.
+The expanded word list is a very large haystack to look for needles in. To make the task somewhat easier you would like to get rid off as much hay as possible, without throwing away any needles. An easy way is to simply drop all words that are spelled correctly. This can be done by filtering the list, excluding all words in a list of correctly spelled words.
 You can get a list of correct words from Aspell. The following command gets a list of Swedish words:

@@ Line 71: / Line 71: @@
 . A new paradigm is needed.
-Take a word at a time. If you have difficulties to figure out the stem of a very strange misspelled word, try to search for it in the original expanded dictionary. You will find the other forms of the word close to the misspelled word. Sometimes this makes it clearer what word it actually should be. If you cannot figure out anyway: go to the next misspelled word.
+Take a word at a time. If you have difficulties to figure out the lexical form of a very strange misspelled word, try to search for it in the original expanded dictionary. You will find the other forms of the word close to the misspelled word. Sometimes this makes it clearer what word it actually should be. If you cannot figure out anyway: go to the next misspelled word.
 There will be plenty of errors to correct. Take the easy ones first!

@@ Line 63: / Line 63: @@
 == Spell-check the rest of the expanded dictionary ==
-The easiest way to quickly find the errors is to check the remaining words in a word processing program of your choice. When you find a misspelled word, try to figure out what's the surface form of the word. Look for it in the Apertium monodix and correct the entry. Very often the error is due to one of the following mistakes:
+The easiest way to quickly find the errors is to check the remaining words in a word processing program of your choice. When you find a misspelled word, try to figure out what's the lexical form of the word. Look for it in the Apertium monodix and correct the entry. Very often the error is due to one of the following mistakes:
 . Wrong stem.

@@ Line 58: / Line 58: @@
 grep -v -wFf  aspellwords.sv > swe.expanded.felstavade
-Now you've got the suspected errors in the file swe.expanded.felstavade (felstavade = misspelled).
+You would find the suspected errors in the file swe.expanded.felstavade (felstavade = misspelled).
 Please change the file names above to what's appropriate in your case.

@@ Line 75: / Line 75: @@
 There will be plenty of errors to correct. Take the easy ones first!
-== Option: check for duplicate entries ==
+== Option: Check for duplicate entries ==
 It might happen that there are duplicate entries for the same word. You can easily find them if you make a frequency list out of the expanded dictionary. This will make a frequency list for the Swedish expanded dictionary: