Task ideas for Google Code-in/Grow bilingual - Revision history

Tino Didriksen: /* How to find the most frequent unknowns */

2020-01-19T15:01:26Z

How to find the most frequent unknowns

Firespeaker at 18:29, 8 November 2019

2019-11-08T18:29:39Z

Firespeaker at 18:29, 8 November 2019

2019-11-08T18:29:22Z

Firespeaker at 18:22, 8 November 2019

2019-11-08T18:22:43Z

Firespeaker at 03:25, 21 December 2017

2017-12-21T03:25:05Z

Firespeaker at 04:51, 14 December 2017

2017-12-14T04:51:29Z

Mlforcada: Created page with "# select a language pair, ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rat..."

2014-11-07T12:07:23Z

Created page with "# select a language pair, ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rat..."

New page

# select a language pair, ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rather good monolingual dictionaries in Apertium but no reasonable bilingual dictionary (these language pairs are usually in the incubator), for instance apertium-spa-pol
# Install Apertium locally from the Subversion repository; install the language pair; make sure that it works and/or get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
# Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) detect the 50 most frequent unknown words (source words which are not in the bilingual dictionaries of the language pair).
# add these correspondences to the bilingual dictionary (so that they are not unknown anymore).
# Compile and test again
# Submit a patch to your mentor (or commit it if you have already gained developer access)

[[Category:Tasks for Google Code-in|Grow bilingual]]

@@ Line 11: / Line 11: @@
            ones with * at the start, sort, count number of hits per word, sort
            again
+<Unhammer> e.g.
-<Unhammer> e.g.                                                         [11:25]
 <Unhammer> zcat corpus.txt.gz | apertium -d . ron-fra | tr ' ' '\n' | grep
-           '^\*' | sort |uniq -d |sort -n >hitlist
+           '^\*' | sort | uniq -c | sort -n >hitlist
-<asusAndrei> awesome!                                                   [11:26]
+<asusAndrei> awesome!
 <Unhammer> hitlist will be unknowns sorted by frequency, but you might have to
            skip a couple that are "strange" or difficult to add

@@ Line 1: / Line 1: @@
 # '''Select a language pair''', ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rather good monolingual dictionaries in Apertium but no reasonable bilingual dictionary (these language pairs are usually in the incubator), for instance apertium-spa-pol
-# '''Install Apertium''' locally from nightlies [[Installation#Installing:_a_summary|instructions here]]; clone the relevant language modules and pair from GitHub; make sure that it works.  Alternatively, get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
+# '''Install Apertium''' locally from nightlies ([[Installation#Installing:_a_summary|instructions here]]); clone the relevant language modules and pair from GitHub; make sure that it works.  Alternatively, get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
 # Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (words in the source document which are not in the bilingual dictionaries of the language pair).  See below for information about how to do this.  Note: the beginner version of this task only requires 50 words.
 # '''Add these correspondences to the bilingual dictionary''' (the appropriate <code>.dix</code> file) in [[bidix]] format (so that they are not unknown anymore), as well as the monolingual analysers if needed.  Make sure to categorise stems correctly.

@@ Line 1: / Line 1: @@
 # '''Select a language pair''', ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rather good monolingual dictionaries in Apertium but no reasonable bilingual dictionary (these language pairs are usually in the incubator), for instance apertium-spa-pol
-# '''Install Apertium''' locally from nightlies; clone the relevant language modules and pair from GitHub; make sure that it works.  Alternatively, get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
+# '''Install Apertium''' locally from nightlies [[Installation#Installing:_a_summary|instructions here]]; clone the relevant language modules and pair from GitHub; make sure that it works.  Alternatively, get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
-# Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (source words which are not  in the bilingual dictionaries of the language pair).  See
+# Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (words in the source document which are not in the bilingual dictionaries of the language pair).  See below for information about how to do this.  Note: the beginner version of this task only requires 50 words.
 # '''Add these correspondences to the bilingual dictionary''' (the appropriate <code>.dix</code> file) in [[bidix]] format (so that they are not unknown anymore), as well as the monolingual analysers if needed.  Make sure to categorise stems correctly.
 # '''Compile and test again'''

@@ Line 1: / Line 1: @@
-# '''select a language pair''', ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rather good monolingual dictionaries in Apertium but no reasonable bilingual dictionary (these language pairs are usually in the incubator), for instance apertium-spa-pol
+# '''Select a language pair''', ideally such that the source language is a language you know (L₂) and the target language a language you use every day (L₁), such that it has rather good monolingual dictionaries in Apertium but no reasonable bilingual dictionary (these language pairs are usually in the incubator), for instance apertium-spa-pol
-# '''Install Apertium''' locally from the Subversion repository; install the language pair; make sure that it works and/or get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
+# '''Install Apertium''' locally from nightlies; clone the relevant language modules and pair from GitHub; make sure that it works.  Alternatively, get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
-# Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (source words which are not  in the bilingual dictionaries of the language pair).
+# Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (source words which are not  in the bilingual dictionaries of the language pair).  See
-# '''add these correspondences to the bilingual dictionary''' (the appropriate <code>.dix</code> file) in [[bidix]] format (so that they are not unknown anymore).
+# '''Add these correspondences to the bilingual dictionary''' (the appropriate <code>.dix</code> file) in [[bidix]] format (so that they are not unknown anymore), as well as the monolingual analysers if needed.  Make sure to categorise stems correctly.
 # '''Compile and test again'''
-# '''Submit''' a patch to your mentor (or commit it if you have already gained developer access)
+# '''Submit''' a pull request to the GitHub repositories
+==How to find the most frequent unknowns==
+<pre>
+<Unhammer> translate your corpus, make it one word per line, grab only the
+           ones with * at the start, sort, count number of hits per word, sort
+           again
+<Unhammer> e.g.                                                         [11:25]
+<Unhammer> zcat corpus.txt.gz | apertium -d . ron-fra | tr ' ' '\n' | grep
+           '^\*' | sort |uniq -d |sort -n >hitlist
+<asusAndrei> awesome!                                                   [11:26]
+<Unhammer> hitlist will be unknowns sorted by frequency, but you might have to
+           skip a couple that are "strange" or difficult to add
+<Unhammer> and that's ok, as long as you start from the most frequent and work
+           your way down
+</pre>
 [[Category:Tasks for Google Code-in|Grow bilingual]]

@@ Line 2: / Line 2: @@
 # '''Install Apertium''' locally from the Subversion repository; install the language pair; make sure that it works and/or get [http://wiki.apertium.org/wiki/Apertium_VirtualBox Apertium VirtualBox] and update, check out & compile the language pair.
 # Using a large enough corpus of representative text in the source language (e.g. plain text taken from Wikipedia, newspapers, literature, etc.) '''detect the 200 most frequent unknown words''' (source words which are not  in the bilingual dictionaries of the language pair).
-# '''add these correspondences to the bilingual dictionary''' (so that they are not unknown anymore).
+# '''add these correspondences to the bilingual dictionary''' (the appropriate <code>.dix</code> file) in [[bidix]] format (so that they are not unknown anymore).
 # '''Compile and test again'''
 # '''Submit''' a patch to your mentor (or commit it if you have already gained developer access)