Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Finding errors in dictionaries

From Apertium
(Difference between revisions)
Jump to: navigation, search
(Exclude the correctly spelled words)
(Exclude the correctly spelled words)
Line 41: Line 41:
 
grep -v -wFf aspellwords.sv > apertium-swe.sv.felstavade
 
grep -v -wFf aspellwords.sv > apertium-swe.sv.felstavade
   
Now you've got the suspected errors in the file apertium-swe.sv.felstavade (felstavade = missspelled).
+
Now you've got the suspected errors in the file apertium-swe.sv.felstavade (felstavade = misspelled).
   
 
== Spell-check the rest of the expanded dictionary ==
 
== Spell-check the rest of the expanded dictionary ==

Revision as of 17:25, 15 February 2015

Contents

Summary

1. Expand the monodix 2. Exclude a list of correctly spelled words 3. Spell-check the rest of the words in a Word processing program of your choice. 4. Edit the monodix for the misspelled words you find.


Expand the monodix

text

Make a list of correctly spelled words

The expanded word list is a very large haystack to look for needles in. To make the task somewhat easier you would like to get rid off as much hay as possible, without throwing away any needles. An easy way is to simply drop all words that are spelled correctly. This can be done by filtering the list against a list of correctly spelled words.

You can get a list of correct words from Aspell. The following command gets a list of English words:

aspell -d en dump master | aspell -l en expand > aspellwords.en

Just change the language code for the language you are working with. For e.g. Swedish it would be:

aspell -d sv dump master | aspell -l sv expand > aspellwords.sv

This list is however rather short. You might find it useful to filter on more words. One way of getting more correctly spelled words is to simply use the top of a word frequency list made on a large corpus. Rational: most people spell correctly most of the time. Highly frequent words are most probably correctly spelled. If they are not they will probably be the new standard for spelling :-)

You can download a corpus from eg. OPUS OPUS Uppsala University. Choose among Europarl, OpenOffice and OpenSubtitles etc in many languages.

You can get a frequency list for instance with the following command:

cat my_english_corpus.txt | tr ' ' '\n' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]' | sort | uniq -c | sort -rn > frequency.en

You can read more about getting a corpus and making a frequency list at the page Building_dictionaries.

Exclude the correctly spelled words

When you've got a nice long list of correctly spelled words to exclude, filter the expanded wordlist from Apertium. This is easy to accomplish with grep. The following command would for instance filter the expanded the Swedish monolingual dictionary:

cat apertium-swe.swe.dix.expanded | grep -v -wFf top_frequency.sv | grep -v -wFf aspellwords.sv > apertium-swe.sv.felstavade

Now you've got the suspected errors in the file apertium-swe.sv.felstavade (felstavade = misspelled).

Spell-check the rest of the expanded dictionary

text


--Tunedal (talk) 16:24, 11 February 2015 (CET)

Personal tools