Ideas for Google Summer of Code/Improved bilingual dictionary induction
Imagine you have two morphological analysers, but no bilingual dictionary. But, you have a parallel corpus. For example: Romanian-French (a French--Romanian parallel corpus can be generated from EuroParl. You can analyse the corpus with a morphological analyser, and use some word-aligner (e.g. GIZA++) to get word alignments.
The problem is that you can't make the bidix entries directly from that. The user will have to specify templates for bidix entries (see for example bilingual dictionary) which map SL-paradigm : TL-paradigm.
When building the bilingual dictionary, any alignment for which the SL word's paradigm doesn't have a template with the TL word's paradigm will be discarded.
- Example
French dictionary:
<e lm="temps"><i>temps</i><par n="mois__n"/></e> temps:temps<n><m><sp>
Romanian dictionary:
<e lm="timp" a="mioara"><i>timp</i><par n="timp__n"/></e> <e lm="vreme" r="LR"><i>vrem</i><par n="vrem/e__n"/></e> timp:timp<n><nt><sg><nom><ind> timpul:timp<n><nt><sg><nom><def> timp:timp<n><nt><sg><dg><ind> timpului:timp<n><nt><sg><dg><def> timpuri:timp<n><nt><pl><nom><ind> timpurile:timp<n><nt><pl><nom><def> timpuri:timp<n><nt><pl><dg><ind> timpurilor:timp<n><nt><pl><dg><def> vreme:vreme<n><f><sg><nom><ind> vremea:vreme<n><f><sg><nom><def> vremi:vreme<n><f><sg><dg><ind> vremii:vreme<n><f><sg><dg><def> vremi:vreme<n><f><pl><nom><ind> vremile:vreme<n><f><pl><nom><def> vremi:vreme<n><f><pl><dg><ind> vremilor:vreme<n><f><pl><dg><def>
Let's suppose we find in the alignments:
temps:timp temps:vreme
We will need patterns to match forms in mois__n
to forms in timp__n
and forms in mois__n
to forms in vrem/e__n
. For example:
<e r="LR"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>timp<s n="n"/><s n="nt"/><s n="ND"/></r></p></e> <e r="RL"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>timp<s n="n"/><s n="nt"/><s n="sg"/></r></p></e> <e r="RL"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>timp<s n="n"/><s n="nt"/><s n="pl"/></r></p></e> <e r="LR"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>vreme<s n="n"/><s n="f"/><s n="ND"/></r></p></e> <e r="RL"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>vreme<s n="n"/><s n="f"/><s n="sg"/></r></p></e> <e r="RL"><p><l>temps<s n="n"/><s n="m"/><s n="sp"/></l><r>vreme<s n="n"/><s n="f"/><s n="pl"/></r></p></e>
There will be a script to extract the most frequent combinations of paradigms in SL-TL, so the user can prioritise which templates to make. So, generating the bidix would be done in an incremental fashion. A lot of the noise of the alignment process can be filtered out by disallowing combinations of words because of no existing paradigm-paradigm model (e.g. mois__n
to cu__pr
)
Tasks
Coding challenge
- Install Apertium
- Install GIZA++
- Generate a word alignment model for a parallel corpus of your choice.
Rewrite the script generate-bidix-templates.py to use python3/ElementTree.- Done here: generate-bidix-templates.py
Frequently asked questions
- none yet, ask us something! :)