Building dictionaries - Revision history

Unhammer: /* Generating bilingual dictionary entries */

2020-03-15T21:36:30Z

Generating bilingual dictionary entries

2018-08-09T10:07:11Z

Frequency

2017-03-31T19:40:09Z

Wikipedia dumps

2016-03-15T07:45:15Z

Generating bilingual dictionary entries

2016-03-15T07:44:54Z

Generating bilingual dictionary entries

2016-03-15T07:39:11Z

Generating bilingual dictionary entries

2016-02-18T09:40:23Z

Generating bilingual dictionary entries

2016-01-30T13:28:43Z

Wikipedia dumps

2015-08-31T11:08:52Z

Generating bilingual dictionary entries

2015-02-27T09:00:57Z

Generating bilingual dictionary entries

← Older revision		Revision as of 21:36, 15 March 2020
Line 270:		Line 270:

	* Use Wiktionary (see below)		* Use Wiktionary (see below)

			* Run [https://research.swtch.com/church-tetragram.pdf word_align] to align words using the tetragrams of a small bilingual corpus ("word_align and char_align were designed to work robustly on texts that are smaller and more noisy than the Hansards")

@@ Line 37: / Line 37: @@
 </pre>
-[I took this from ''Unix for Poets'', I think.]
+[I took this from ''Unix for Poets'', I think. More info on making frequency lists at [[Make a frequency list]].)
 Of course, this may be improved a lot but serves for illustration

@@ Line 110: / Line 110: @@
 For help in processing them, see:
-* http://meta.wikimedia.org/wiki/Help:Export
+* http://www.mediawiki.org/wiki/Help:Export
 The dumps need cleaning up (removing Wiki syntax and XML etc.), but can

@@ Line 243: / Line 243: @@
 * Run [[ReTraTos]] on word alignments if you have a parallel corpus
 ** or just [[Giza]] and look through the symmetrised dictionary created; see [[Extracting bilingual dictionaries with Giza++]]
-** or just [[Anymalign]] / [[Aligning_a_corpus_with_fast_align|fast align]] and look through the generated word alignments
+** or just [[Anymalign]] / [[Aligning_a_corpus_with_fast_align|fast_align]] and look through the generated word alignments
 * Spell a source language frequency list using a target-language spell checker

@@ Line 243: / Line 243: @@
 * Run [[ReTraTos]] on word alignments if you have a parallel corpus
 ** or just [[Giza]] and look through the symmetrised dictionary created; see [[Extracting bilingual dictionaries with Giza++]]
-** or just [[Anymalign]] and look through the generated word alignments
+** or just [[Anymalign]] / [[Aligning_a_corpus_with_fast_align|fast align]] and look through the generated word alignments
 * Spell a source language frequency list using a target-language spell checker

@@ Line 238: / Line 238: @@
 * Make a TMX with [[bitextor]]
+* Grab some translations from [[Wikidata]]
 * Run [[ReTraTos]] on word alignments if you have a parallel corpus

@@ Line 122: / Line 122: @@
 content, and is the archive most mirror sites will probably want."
+Use one of the methods on [[Wikipedia dumps]] to clean your corpus.
-Something like (for Afrikaans):
+Or a simple shell script like (for Afrikaans):
 <pre>

@@ Line 250: / Line 250: @@
 ** See [[Automatically generating compound bidix entries]] for one approach, scoring with an LM
 ** Maybe decompound your ''bidix'' as well (requires an analyser that can give compound analyses even if full analyses exist) and use that to create a bigger compound-part-bidix to run the above procedure on
+* Cluster compound-parts to find translations of parts through compounds with other parts
+** Say you have big monolingual dictionaries and want to expand the bidix. E.g. the sme word bearaš is missing in sme-nob, but it appears as a prefix in lots of monolingual compounds, e.g. bearašviessu, bearašásodat. Many of the _suffixes_ are in the bidix. So look up these suffixes, and find which nob-words take the translations of the suffixes. The ones that take the most of these suffix-translations are likely candidates for a prefix-translation.
 * Generate possible Multiword Compounds from existing multiword compounds, check for them in corpus