Finding errors in dictionaries
Contents
Summary
1. Expand the monodix 2. Exclude a list of correctly spelled words 3. Spell-check the rest of the words in a Word processing program of your choice. 4. Edit the monodix for the misspelled words you find.
Expand the monodix
text
Make a list of correctly spelled words
The expanded word list is a very large haystack to look for needles in. To make the task somewhat easier you would like to get rid off as much hay as possible, without throwing away any needles. An easy way is to simply drop all words that are spelled correctly. This can be done by filtering the list against a list of correctly spelled words.
You can get a list of correct words from Aspell. The following command gets a list of English words:
aspell -d en dump master | aspell -l en expand > aspellwords.en
Just change the language code for the language you are working with. For e.g. Swedish it would be:
aspell -d sv dump master | aspell -l sv expand > aspellwords.sv
This list is however rather short. You might find it useful to filter on more words. One way of getting more correctly spelled words is to simply use the top of a word frequency list made on a large corpus. Rational: most people spell correctly most of the time. Highly frequent words are most probably correctly spelled. If they are not they will probably be the new standard for spelling :-)
You can download a corpus from eg. OPUS OPUS Uppsala University. Choose among Europarl, OpenOffice and OpenSubtitles etc in many languages.
You can get a frequency list for instance with the following command:
cat my_english_corpus.txt | tr ' ' '\n' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]' | sort | uniq -c | sort -rn > frequency.en
You can read more about getting a corpus and making a frequency list at the page Building_dictionaries.
Exclude the correctly spelled words
When you've got a nice long list of correctly spelled words to exclude, filter the expanded wordlist from Apertium. This is easy to accomplish with grep. The following command would for instance filter the expanded the Swedish monolingual dictionary:
cat apertium-swe.swe.dix.expanded | grep -v -wFf top_frequency.sv | grep -v -wFf aspellwords.sv > apertium-swe.sv.felstavade
Now you've got the suspected errors in the file apertium-swe.sv.felstavade (felstavade = missspelled).
Spell-check the rest of the expanded dictionary
text
