User:Xavivars/Overlapping language variants
Contents
Problem
Language variants are a great feature of Apertium. It allows you to create slightly different language pairs, where the core of the translations are the same.
However, the way variants are implemented is not properly suited for overlapping variants, when we have more than two variants for the same language that are not completely disjoint.
Let's take Catalan as an example:
- Currently, we have the base Catalan translation (cat), plus a Valencian variant (cat_valencia) based on the Universities language model.
- There used to exist an "economic focused" variant (cat_eco).
- Work is being done by Joan Moratinos to add suppor for a balearic variant (cat_balear, temptative name).
Let's imagine that we want to add another Valencian variant, based on the Generalitat (Valencian Government) or the AVL (Valencian Language Accademy), (cat_gva or cat_avl are the names I'll use here to support the example).
Let's also define a set of "language features", and are not overlapping:
- Verbs
- CAT verb terminations (penso)
- VAL verb terminations (pense)
- BAL verb terminations (pens)
- Lexical choice
- CAT lexical (sortir)
- VAL lexical (eixir)
- Accents
- CAT accents (interès)
- VAL accents (interés)
- Demonstratives
- Weak (aquest)
- Strong (este)
- Possessives
- U (meua, teua)
- V (meva, teva)
- Economics lexic
- Generic
- Economics focused
The problem comes on how the different variants use each of the language features. As an example, the following chart shows only two of the features: verbs and accents.
For this example, if we just keep the current variants approach, we would need four variants to be present in the dictionaries. All 4 variants would need to define their own version for each "feature", (even if some of them are identical):
cat_balear
would need to have all the same verb entries thancat
, and same thing would happen forcat_valencia
andcat_gva
.- In case of accents,
cat
,cat_balear
andcat_valencia
, as they share the same accents, would create the same entries three times in the dictionary, while another different entry would be added to supportcat_gva
Proposed approaches
Metadix and variant definition
This is a very naive approach, but could work without much involvement for Apertium as a framework.
The idea would be to create a new "metadix" for that language (and language pairs that use it) so when building the pair, a proper dix gets created. In the metadix, instead of current variants, "language features" will be tagged. And variants will be defined in a "config" file as a set of language features.
So, as an example, instead of
<e v="cat"><p><l>la<b/>meva</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> <e v="val"><p><l>la<b/>meua</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> ... <e> <p><l>ès</l> <r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e> <e r="LR"><p><l>és</l> <r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e>
we will have something closer to
... <e lf="possessive-v"><p><l>la<b/>meva</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> <e lf="possessive-u"><p><l>la<b/>meua</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> ... <e lf="accents-cat"><p><l>ès</l><r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e> <e lf="accents-val"><p><l>és</l><r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e> ...
and a config file like this one (expressed in json, but can be anything)
{ groups : { "possessive" : ["possesive-v", "possessive-u"], "accents" : ["accents-val", "accents-cat"], ... }, variants : { cat: { "possessive-v", "accents-cat", "verbs-cat" }, cat_valencia: { "possessive-v", "accents-cat", "verbs-val" }, cat_balear: { "possessive-v", "accents-cat", "verbs-bal" }, cat_gva:{ "possessive-u", "accents-val", "verbs-val" } } }
- Seems like a good solution! If you make the config XML too, I think it'd be quite possible to do using xsltproc, as with the other metadix features. --unhammer (talk) 20:16, 19 February 2017 (CET)
Id-based variants
The idea is to have entries that are alternative language variant forms to one "general" form marked with a unique ID as follows:
<e eid="la-meua"><p><l>la<b/>meva</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> ... <e eid="è-n"><p><l>è</l><r>è<s n="n"/><s n="m"/><s n="sg"/></r></p></e> ... <e><i>poal</i><pardef n="abric__n"></e> <e><i>galleda</i><pardef n="abric__n"></e>
In the bilingual dictionary we have:
... <e><p><l>cubo<s n="n"/><s n="m"/></l><r>galleda<s n="n"/><s n="f"/></r></p></e> ...
In a different file, we explicit what we do to the alternatives marked with ID by variant with sections like
<?xml version=“1.0”?> <variant name=“valencia”> <replace> <e eid="la-meva"><p><l>la<b/>meua</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> </replace> <replace> <e id="café-nsg"><p><l>é</l> <r>è<s n="n"/><s n="m"/><s n="sg"/></r></p></e> <e> <p><l>è</l> <r>è<s n="n"/><s n="m"/><s n="sg"/></r></p></e> </replace> <add> </add> <remove> </remove> </variant>
In this file, we maintain all the info related to variants.
One entry, multiple variants
The idea is to extend the attribute v for entries so as it can carry more than one value:
<e v="cat"><p><l>la<b/>meva</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> <e v="cat_uni cat_gva"><p><l>la<b/>meua</l><r>el<b/>meu<s n="det"/><s n="pos"/><s n="f"/><s n="sg"/></r></p></e> ... <e v="cat cat_uni> <p><l>ès</l> <r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e> <e v="cat_gva"> <p><l>és</l> <r>ès<s n="n"/><s n="m"/><s n="sg"/></r></p></e> ... <e v="cat_uni cat_gva"><i>poal</i><pardef n="abric__n"></e> <e v="cat"><i>galleda</i><pardef n="abric__n"></e> ...
As we do now, we would create the appropriate entries for generation depending on the variant during compilation in the Makefile
... VAR1=#deixar en blanc VAR2=_valencia_uni VAR3=_valencia_gva ... $(LANG1)$(VAR1).autogen.bin: $(BASENAME).$(LANG1).dix .deps/.d apertium-validate-dictionary $< lt-comp -c cat rl $< $@ $(BASENAME).$(LANG1).acx $(LANG1)$(VAR2).autogen.bin: $(BASENAME).$(LANG1).dix .deps/.d apertium-validate-dictionary $< lt-comp -c cat_uni rl $< $@ $(BASENAME).$(LANG1).acx $(LANG1)$(VAR3).autogen.bin: $(BASENAME).$(LANG1).dix .deps/.d apertium-validate-dictionary $< lt-comp -c cat_gva rl $< $@ $(BASENAME).$(LANG1).acx
Benefits: it is simple, all the info stays in a single dix file and is easy to understand.