Difference between revisions of "Compounds"

From Apertium
Jump to navigation Jump to search
 
(54 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 
{{TOCD}}
 
{{TOCD}}
Some languages (in Indo-European particularly Germanic languages) like to make long compound words with low frequency that are unlikely to be found in dictionaries. Typically for any "normal" noun, there can be around 10—100 compound nouns which inflect in exactly the same way (at least for Afrikaans).
+
Some languages in Indo-European, particularly [[Germanic language]]s and Proto-Indo-Iranian languages like [http://wiki.apertium.org/wiki/Sanskrit Sanskrit], make long compound words with low frequency that are unlikely to be found in dictionaries. Typically for any "normal" noun, there can be around 10—100 compound nouns which inflect in exactly the same way (at least for Afrikaans).
   
* Afrikaans: infrastruktuurontwikkelingsplan, infrastruktuur+ontwikkelings+plan ("infrastructure development plan"), (cf. personeelverminderingsprosedure, "personnel protection procedure")
+
* [[Afrikaans]]: infrastruktuurontwikkelingsplan, infrastruktuur+ontwikkelings+plan ("infrastructure development plan"), (cf. personeelverminderingsprosedure, "personnel protection procedure")
* Dutch : "hulpagina" (help page), "woordbetekenis" (meaning of a word), "inwonertal" (number of inhabitants)
+
* [[Dutch]] : "hulpagina" (help page), "woordbetekenis" (meaning of a word), "inwonertal" (number of inhabitants)
* German: Kontaktlinsenverträglichkeitstest, Kontakt+linsen+verträglichkeits+test ("contact-lens compatibility test")
+
* [[German]]: Kontaktlinsenverträglichkeitstest, Kontakt+linsen+verträglichkeits+test ("contact-lens compatibility test")
* Danish: Kontaktlinsevæske, Kontaktlinse+væske ("contact-lens liquid")
+
* [[Danish]]: Kontaktlinsevæske, Kontaktlinse+væske ("contact-lens liquid")
  +
* [[Sanskrit]]: विद्या + आतुर = विद्यातुर , vidyā + ātur = vidyātur ("eager to gain knowledge")
  +
* [[Esperanto]]: Vikitraduko, Vikio+traduko (Wiki translation). Tradukoservo (==tradukservo), traduko+servo (translation service). Poŝtelefono, poŝa+telefono (pocket phone). Bonkvalita, bona+kvalita (good quality). Diaro, dio+aro (god collection),
   
The presence or absence of a connecting morpheme can sometimes signal a semantic difference, compare:
 
* ''sjefsekretær'' = 'chief of secretaries' (Norwegian)
 
* ''sjefssekretær'' = 'the boss's secretary'
 
   
There should be some method of attempting to resolve unknown compound words into their constituent parts.
+
Both [[lttoolbox]] and [[HFST]] have methods for dynamically analysing unknown compound words into their constituent parts. See below for how it's done in lttoolbox.
   
  +
==Simple lttoolbox example==
See also [http://bugs.apertium.org/cgi-bin/bugzilla/show_bug.cgi?id=13 the bug report]
 
==Outstanding questions==
 
   
  +
In Norwegian, we ''generally'' want nouns to be possible non-final parts of compounds when they are in "lemma form", while they can be final parts in all forms (similar to how you can say "bookshops" but not "booksshop" – we will ignore exceptions for now). Say we have a noun paradigm <code>ep__n</code> that looks like
* Where would compound processing go in the pipeline? Presumably after initial analysis? e.g. in between <code>lt-proc</code> and <code>apertium-tagger</code>.
 
  +
  +
<pre>
  +
<pardef n="ep__n">
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p></e>
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p></e>
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p></e>
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p></e>
  +
</pardef>
  +
</pre>
   
  +
and some words that use this are "snikmordar" (assassin) and "tørrfisk" (stockfish). Now someone writes a book about a stockfish assassin and we have to analyse that.
==Proposed algorithms==
 
   
  +
We'll add some helper paradigms, to avoid repetition:
===Analysis===
 
   
  +
<pre>
;Baseline
 
  +
<pardef n="cp-R"
  +
c="Add hidden tag during analysis (allowing use as final compound part),
  +
allow generation without the tag.">
  +
<e r="RL"><p><l></l> <r></r></p></e>
  +
<e r="LR"><p><l></l> <r><s n="compound-R"/></r></p></e>
  +
</pardef>
  +
  +
<pardef n="cp-L"
  +
c="Add hidden tag for analysis as non-final compound (along with non-hidden cmp-tag),
  +
allow generation with non-hidden cmp-tag">
  +
<e r="RL"><p><l></l> <r><s n="cmp"/></r></p></e>
  +
<e r="LR"><p><l></l> <r><s n="cmp"/><s n="compound-only-L"/></r></p></e>
  +
</pardef>
  +
  +
<pardef n="cp-both"
  +
c="For analyses that can be both non-final and final">
  +
<e> <p><l></l> <r></r></p> <par n="cp-L"/></e>
  +
<e> <p><l></l> <r></r></p> <par n="cp-R"/></e>
  +
</pardef>
  +
</pre>
   
  +
And we use the helpers like this:
input: ^*infrastruktuurontwikkelingsplan$
 
   
  +
<pre>
# Read word from left to right.
 
  +
<pardef n="ep__n">
# Take the shortest match first from the dictionaries, e.g. infrastruktuurontwikkelingsplan,
 
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p><par n="cp-R"/></e>
## Read i-n-f-r-a-s-t-r-u-k-u-u-r (add, because no words have +o)
 
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p><par n="cp-R"/></e>
## Read o-n-t-w-i-k-k-e-l-i-n-g-s (add, because no words have +p)
 
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p><par n="cp-R"/></e>
## Read p-l-a-n
 
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="cp-both"/></e>
# Output in order.
 
  +
</pardef>
   
  +
output: ^infrastruktuur<n><sg>$ ^ontwikkeling<n><pl>$ ^plan<n><sg>$
 
   
  +
<e> <i>tørrfisk</i> <par n="ep__n"/></e>
;Left-to-right longest-match
 
  +
<e> <i>snikmordar</i><par n="ep__n"/></e>
  +
</pre>
   
  +
Using <code>lt-proc</code> with the <code>-e</code> switch we turn on dynamic compounding:
input: ^*infrastruktuurontwikkelingsplan$
 
   
  +
<pre>
# Read word from left to right.
 
  +
$ lt-comp lr cmp.dix cmp.bin
# Take the longest match first from the dictionaries, e.g. infrastruktuurontwikkelingsplan,
 
  +
main@standard 32 36
# While not found:
 
  +
$ printf "%s\n" tørrfisk tørrfisken snikmordar tørrfisksnikmordar tørrfisksnikmordaren tørrfiskensnikmordar | lt-proc -we cmp.bin
## Read infrastruktuurontwikkelingsplan
 
  +
^tørrfisk/tørrfisk<n><m><sg><ind>$
## Read infrastruktuurontwikkelingspla
 
  +
^tørrfisken/tørrfisk<n><m><sg><def>$
## Read infrastruktuurontwikkelingspl
 
  +
^snikmordar/snikmordar<n><m><sg><ind>$
## ...
 
  +
^tørrfisksnikmordar/tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><ind>$
# When a word is found, e.g. "infrastruktuur", remove it from the string, put it in the output queue, and start the process again with ontwikkelingsplan.
 
  +
^tørrfisksnikmordaren/tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><def>$
# Output each item from the output queue in order.
 
  +
^tørrfiskensnikmordar/*tørrfiskensnikmordar$
  +
</pre>
  +
Note how we don't allow inflected non-final parts (<code>tørrfisk&lt;def&gt;+snikmordar</code>).
   
  +
lttoolbox will also allow longer strings of (non-final)+ final
output: ^infrastruktuur<n><sg>$ ^ontwikkeling<n><pl>$ ^plan<n><sg>$
 
  +
<pre>
  +
^snikmordartørrfisksnikmordaren/snikmordar<n><m><sg><ind><cmp>+tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><def>$
  +
</pre>
  +
There is no dynamic compounding magic for generation – transfer should simply output several lexical units in a row with no spaces between them. We typically use a (non-hidden) &lt;cmp&gt; tag for the non-final parts in case they differ from the "uninflected" form
  +
<pre>
  +
$ lt-comp rl cmp.dix cmp.gin
  +
main@standard 30 33
  +
$ echo '^tørrfisk<n><m><sg><ind><cmp>$^snikmordar<n><m><sg><def>$' | lt-proc -g cmp.gin
  +
tørrfisksnikmordaren
  +
</pre>
   
  +
==Differences in left-parts and right-parts of compounds==
===Generation===
 
  +
In Germanic languages at least (possibly most others too?), compounds typically only inflect in the last part:
  +
* mobiltelefon = cell phone
  +
* mobiltelefonar = cell phones
  +
* NOT: *mobilartelefon ("cells phone")
   
  +
The way this is solved in the current compound implementation in [[lttoolbox]] and [[lttoolbox-java]] is to have one symbol for word forms that may be a non-final part of a compund, <code>compound-only-L</code>, and one for forms that may end a compound, <code>compound-R</code>:
If we have translated a string of input words from English to Afrikaans and now want to compound the ones that can be compounded, we have a problem that instead of:
 
   
  +
<pre>
:infrastruktuurontwikkelingsplan
 
  +
<pardef n="ep__n">
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-R"/></r></p></e>
  +
<e r="LR"><p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
  +
</pardef>
  +
...
  +
<e lm="mobil"><i>mobil</i><par n="ep__n"/></e>
  +
<e lm="telefon"><i>telefon</i><par n="ep__n"/></e>
  +
</pre>
   
  +
The "only" in here <code>compound-only-L</code> means that this form may '''only''' appear as part of a compound, not alone. The reason for this is explained [[Compounds#Epenthetics|below]].
we have:
 
   
  +
Note: the compound symbols do '''not''' appear in the output, they only signal to lt-proc that this form may appear in compounds.
:infrastruktuur ontwikkeling plan
 
   
  +
==Limitations of lttoolbox compounding==
One way of resolving this would be to compile a wordlist (without morphological info) into a tree, and then scan the tree as we scan the output, concatenating words where they appear in the wordlist. Wordlists are substantially more numerous than morphological resources so it would be quite cheap. In order to avoid false-positives, we could just set a length requirement of something like 9 characters or more.
 
   
  +
The lttoolbox method is to first try analyzing a token without compounding, and only do compounding if the other methods would give an unknown word. Unknown words are made up of strings of characters from &lt;alphabet&gt;, separated by non-alphabetics or inconditional analyses.
==Further reading==
 
   
  +
So if the dictionary contains
  +
<pre>
  +
<alphabet>abcdefghijklmnopqrstuvwxyz01-</alphabet>
  +
  +
<section id="main" type="standard">
  +
<e> <p><l>oste</l> <r>ost<s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
  +
<e> <p><l>kaker</l> <r>kake<s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l>100-</l> <r>100<s n="det"/><s n="compound-only-L"/></r></p></e>
  +
<e> <p><l>200-</l> <r>200<s n="det"/><s n="compound-only-L"/></r></p></e>
  +
</section>
  +
</pre>
  +
Then you'll get compound analyses of "ostekake" and "100-kake" but not "200-kake" ("2" is missing from alphabet).
  +
  +
But if you have an inconditional analysis of numbers and dashes like
  +
<pre>
  +
<section id="main" type="inconditional">
  +
<e> <re>[0-9]+</re><p><l></l> <r><s n="det"/><s n="qnt"/><s n="pl"/></r></p></e>
  +
<e> <p><l>-</l> <r>-<s n="guio"/></r></p></e>
  +
</section>
  +
</pre>
  +
then you won't even get "100-kake", it'll become <pre>^100<det>$^-<guio>$^kake<n>$</pre>
  +
  +
==Epenthetics==
  +
Many compounds have [http://en.wikipedia.org/wiki/Epenthetic epenthetics], letters which connect the two words but perhaps only occur in compounds:
  +
* ''kransekake'' => ''krans+e+kake'' ≈ 'ring cake'
  +
* ''ungdomsfyll'' => ''ungdom+s+fyll'' = 'youth drunkenness'
  +
  +
The epenthetics do not have a meaning in themselves, and can typically only appear between compound parts (not when the word is by itself). (Their existance is purely phonological, "because nobody wants awkward morphemes"[http://linguisticmystic.com/2011/04/09/phonology-is-a-lot-like-high-school-really/].)
  +
  +
===Epenthetic analysis===
  +
We solve the analysis in [[lttoolbox]] and [[lttoolbox-java]] by the following type of paradigm:
  +
<pre>
  +
<pardef n="krans__n">
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/><s n="compound-R"/></r></p></e>
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-R"/></r></p></e>
  +
<e r="LR"><p><l>e</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
  +
</pardef>
  +
</pre>
  +
  +
As noted above, we have a difference between
  +
* <code>compound-R</code> meaning "this might be the right part of a compound, but might also stand alone", and
  +
* <code>compound-only-L</code> meaning "this might be the left part of a compound, but ''can't'' stand alone".
  +
  +
The above paradigm specifies that "kranse" may only appear as a left-part of a compound, it can't stand alone.
  +
  +
: Objection 1: why not have just compound-L and compound-R, meaning "might be the left/right part or might stand alone"? Because you're never going to see "kakee" on its own, right?
  +
:: Answer: True, we won't see "kakee" on its own, but "kranse" is a verb with another meaning, and tagging the noun as "compound-L" and allowing it to be analysed on its own would lead to unnecessary ambiguity.
  +
: Objection 2: Hey, you've got compound-only-L but not compound-L and not compund-only-R. You can't analyse Klingon!
  +
:: Answer: This isn't even implemented yet. Once two symbols are implemented and working, we can think about adding the rest of the possibilites.
  +
: Objection 3: You really should have some sort of compound-both symbol for your <code><l/></code> entry instead of one with compound-only-L and one with compound-R, it'll mess up your tagger.
  +
:: Answer: These symbols are removed in the output.
  +
  +
===Epenthetic generation===
  +
When generating a compound, [[apertium-nn-nb]] at least simply adds the tag <code><cmp></code> to the left-part. The generator can then tell whether the left-part should have an epenthetic or not.
  +
  +
Example: after transfer we might have <code>^krans<n><m><sg><ind><cmp>$^kake<n><f><sg><ind>$</code>. Then the generator just needs a pardef for <code>krans</code> that has the line
  +
<pre>
  +
<e r="RL"><p><l>e</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="cmp"/></r></p></e>
  +
</pre>
  +
in order to output the correct <code>kransekake</code> (epenthetic e).
  +
  +
If the output from transfer is <code>^gulrot<n><m><sg><ind><cmp>$^kake<n><f><sg><ind>$</code> ("carrot cake"), and the generator has
  +
<pre>
  +
<e r="RL"><p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="cmp"/></r></p></e>
  +
</pre>
  +
in the pardef for <code>gulrot</code>, we get the correct <code>gulrotkake</code> (no epenthetic).
  +
  +
===Meaning of epenthetics===
  +
The epenthetic has no meaning in itself, but if you can have the same word with or without an epenthetic, it might signal a difference. Compare:
  +
* ''sjefsekretær'' = 'chief of secretaries' (Norwegian)
  +
* ''sjef'''s'''sekretær'' = 'the boss's secretary'
  +
* ''gjestfri'' = 'hospitable'
  +
* ''gjest'''e'''fri'' = 'guest free' (as in "guest free zone")
  +
(''gjestfri'' would typically be listed in the dictionary, ''gjest'''e'''fri'' being compositional.)
  +
  +
An epenthetic can also determine the meaning of an already ambiguous word-part:
  +
* ''tredørsvarianten'' = 'the three-door model'
  +
* ''tredørvarianten'' = 'the type of the wooden door'
  +
  +
Solution: add these to your dictionaries :-)
  +
  +
==Outstanding questions==
  +
===Stream symbol===
  +
We could use ~ instead of +, this would allow pretransfer to output '^$' instead of '^ $' for compounds (so transfer would no longer have to delete spaces)
  +
  +
: This would need a small change in the tagger too. Can anyone think of a situation where it would need to be treated differently to normal [[def-mult|def-mults]]?
  +
  +
===Dix format===
  +
Currently we use
  +
<pre>
  +
<s n="compound-only-L"/>
  +
<s n="compound-R"/>
  +
</pre>
  +
this could be changed to
  +
<pre>
  +
<c r="L"/>
  +
<c r="R"/>
  +
</pre>
  +
  +
: We could also change "R" to only mark compound paths (so it would mean "compound-only-R"), since having a path that's both a compound part and usable as a full word is [http://apertium.codepad.org/yZnlY6M8 easily done with a pardef]
  +
  +
: Changes to dix format should be coordinated with [[Unification of metadix and parametrized dictionaries|unifying the metadix format]]
  +
  +
===German/Capitalised compounds vs (proper noun) regexes===
  +
Some pairs use a regex to catch proper nouns, acronyms etc. Since regexes are no different from other entries in the FST, these will match before we get a chance to test for compounding. This means German nouns or capitalised compounds in other languages will never get a decompounding analysis.
  +
  +
==When to compound==
  +
  +
You shouldn't allow compounding on all words. In particular, if you allow compounding on all short words, you will get horrible results:
  +
<pre>$ echo bildreportagen |apertium -d . swe-dan
  +
billede #rids mugmide tagene</pre>
  +
  +
In Scandinavian dixen, it's common to have two versions of pardefs, one with and one without compounding:
  +
<pre>
  +
<pardef n="ep__n">
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p><par n="cp-R"/></e>
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p><par n="cp-R"/></e>
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p><par n="cp-R"/></e>
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="cp-both\Ø_LR_s"/></e>
  +
</pardef>
  +
<pardef n="ep_no-cp__n">
  +
<e> <p><l>ane</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p></e>
  +
<e> <p><l>ar</l> <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p></e>
  +
<e> <p><l>en</l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p></e>
  +
<e> <p><l></l> <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="no-cp\Ø"/></e>
  +
</pardef>
  +
</pre>
  +
(where cp-R adds compound-R, cp-both\Ø_LR_s adds compound-only-L with no epenthetic, LR with s epenthetic, and compound-R)
  +
  +
==See also==
  +
* [[Conjoined lexical units]]
  +
  +
==Further reading==
  +
* http://en.wikipedia.org/wiki/Compound_(linguistics)
 
* Koehn, P. and Knight, K. (2003) "[http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/compound2003.pdf Empirical Methods for Compound Splitting]". ''11th Conference of the European Chapter of the Association for Computational Linguistics'', (EACL2003).
 
* Koehn, P. and Knight, K. (2003) "[http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/compound2003.pdf Empirical Methods for Compound Splitting]". ''11th Conference of the European Chapter of the Association for Computational Linguistics'', (EACL2003).
 
* Brown, R. (2002) "[http://www.eamt.org/archive/tmi2002/conference/02_brown.pdf Corpus-Driven Splitting of Compound Words]". ''TMI 2002''
 
* Brown, R. (2002) "[http://www.eamt.org/archive/tmi2002/conference/02_brown.pdf Corpus-Driven Splitting of Compound Words]". ''TMI 2002''
 
* Larson, M., Willett, D., Köhler, J. and Rigoll, G. (2000) "[http://citeseer.ist.psu.edu/cache/papers/cs/15664/http:zSzzSzwww.fb9-ti.uni-duisburg.dezSzpublzSz00zSzicslp00ml_compounds.pdf/larson00compound.pdf Compound splitting and lexical unit recombination for improved performance of a speech recognition system for German parliamentary speeches]". ''Conference on Spoken Language Processing'', 2000.
 
* Larson, M., Willett, D., Köhler, J. and Rigoll, G. (2000) "[http://citeseer.ist.psu.edu/cache/papers/cs/15664/http:zSzzSzwww.fb9-ti.uni-duisburg.dezSzpublzSz00zSzicslp00ml_compounds.pdf/larson00compound.pdf Compound splitting and lexical unit recombination for improved performance of a speech recognition system for German parliamentary speeches]". ''Conference on Spoken Language Processing'', 2000.
 
* Moa, H. (2005) "[http://phon.joensuu.fi/lingjoy/01/moaF.pdf Compounds and other oddities in machine translation]". ''Proceedings of the 15th NODALIDA conference, Joensuu 2005''.
 
* Moa, H. (2005) "[http://phon.joensuu.fi/lingjoy/01/moaF.pdf Compounds and other oddities in machine translation]". ''Proceedings of the 15th NODALIDA conference, Joensuu 2005''.
  +
* Johannesen, J. B. and Hauglin, H. (1996) "[http://folk.uio.no/jannebj/Scan.Conf.Turku-96.ps An automatic analysis of Norwegian compounds]". In Haukioja, T. (ed.): ''Papers from the 16th Scandinavian Conference of Linguistics'', Turku/Åbo, Finland 1996: 209-220.
   
 
[[Category:Development]]
 
[[Category:Development]]
  +
[[Category:Documentation in English]]

Latest revision as of 09:07, 6 April 2021

Some languages in Indo-European, particularly Germanic languages and Proto-Indo-Iranian languages like Sanskrit, make long compound words with low frequency that are unlikely to be found in dictionaries. Typically for any "normal" noun, there can be around 10—100 compound nouns which inflect in exactly the same way (at least for Afrikaans).

  • Afrikaans: infrastruktuurontwikkelingsplan, infrastruktuur+ontwikkelings+plan ("infrastructure development plan"), (cf. personeelverminderingsprosedure, "personnel protection procedure")
  • Dutch : "hulpagina" (help page), "woordbetekenis" (meaning of a word), "inwonertal" (number of inhabitants)
  • German: Kontaktlinsenverträglichkeitstest, Kontakt+linsen+verträglichkeits+test ("contact-lens compatibility test")
  • Danish: Kontaktlinsevæske, Kontaktlinse+væske ("contact-lens liquid")
  • Sanskrit: विद्या + आतुर = विद्यातुर , vidyā + ātur = vidyātur ("eager to gain knowledge")
  • Esperanto: Vikitraduko, Vikio+traduko (Wiki translation). Tradukoservo (==tradukservo), traduko+servo (translation service). Poŝtelefono, poŝa+telefono (pocket phone). Bonkvalita, bona+kvalita (good quality). Diaro, dio+aro (god collection),


Both lttoolbox and HFST have methods for dynamically analysing unknown compound words into their constituent parts. See below for how it's done in lttoolbox.

Simple lttoolbox example[edit]

In Norwegian, we generally want nouns to be possible non-final parts of compounds when they are in "lemma form", while they can be final parts in all forms (similar to how you can say "bookshops" but not "booksshop" – we will ignore exceptions for now). Say we have a noun paradigm ep__n that looks like

        <pardef n="ep__n">
          <e>       <p><l>ane</l>   <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p></e>
          <e>       <p><l>ar</l>    <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p></e>
          <e>       <p><l>en</l>    <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p></e>
          <e>       <p><l></l>      <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p></e>
        </pardef>

and some words that use this are "snikmordar" (assassin) and "tørrfisk" (stockfish). Now someone writes a book about a stockfish assassin and we have to analyse that.

We'll add some helper paradigms, to avoid repetition:

    <pardef n="cp-R"
            c="Add hidden tag during analysis (allowing use as final compound part),
               allow generation without the tag.">
      <e r="RL"><p><l></l>            <r></r></p></e>
      <e r="LR"><p><l></l>            <r><s n="compound-R"/></r></p></e>
    </pardef>
    
    <pardef n="cp-L"
            c="Add hidden tag for analysis as non-final compound (along with non-hidden cmp-tag),
               allow generation with non-hidden cmp-tag">
      <e r="RL"><p><l></l>            <r><s n="cmp"/></r></p></e>
      <e r="LR"><p><l></l>            <r><s n="cmp"/><s n="compound-only-L"/></r></p></e>
    </pardef>
    
    <pardef n="cp-both"
            c="For analyses that can be both non-final and final">
      <e>       <p><l></l>            <r></r></p> <par n="cp-L"/></e>
      <e>       <p><l></l>            <r></r></p> <par n="cp-R"/></e>
    </pardef>

And we use the helpers like this:

    <pardef n="ep__n">
      <e>       <p><l>ane</l>   <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p><par n="cp-R"/></e>
      <e>       <p><l>ar</l>    <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p><par n="cp-R"/></e>
      <e>       <p><l>en</l>    <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p><par n="cp-R"/></e>
      <e>       <p><l></l>      <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="cp-both"/></e>
    </pardef>

    …

    <e>       <i>tørrfisk</i>  <par n="ep__n"/></e>
    <e>       <i>snikmordar</i><par n="ep__n"/></e>

Using lt-proc with the -e switch we turn on dynamic compounding:

    $ lt-comp lr cmp.dix cmp.bin
    main@standard 32 36
    $ printf "%s\n" tørrfisk tørrfisken snikmordar tørrfisksnikmordar tørrfisksnikmordaren tørrfiskensnikmordar | lt-proc -we cmp.bin
    ^tørrfisk/tørrfisk<n><m><sg><ind>$
    ^tørrfisken/tørrfisk<n><m><sg><def>$
    ^snikmordar/snikmordar<n><m><sg><ind>$
    ^tørrfisksnikmordar/tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><ind>$
    ^tørrfisksnikmordaren/tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><def>$
    ^tørrfiskensnikmordar/*tørrfiskensnikmordar$

Note how we don't allow inflected non-final parts (tørrfisk<def>+snikmordar).

lttoolbox will also allow longer strings of (non-final)+ final

    ^snikmordartørrfisksnikmordaren/snikmordar<n><m><sg><ind><cmp>+tørrfisk<n><m><sg><ind><cmp>+snikmordar<n><m><sg><def>$

There is no dynamic compounding magic for generation – transfer should simply output several lexical units in a row with no spaces between them. We typically use a (non-hidden) <cmp> tag for the non-final parts in case they differ from the "uninflected" form

    $ lt-comp rl cmp.dix cmp.gin
    main@standard 30 33
    $ echo '^tørrfisk<n><m><sg><ind><cmp>$^snikmordar<n><m><sg><def>$' | lt-proc -g cmp.gin
    tørrfisksnikmordaren

Differences in left-parts and right-parts of compounds[edit]

In Germanic languages at least (possibly most others too?), compounds typically only inflect in the last part:

  • mobiltelefon = cell phone
  • mobiltelefonar = cell phones
  • NOT: *mobilartelefon ("cells phone")

The way this is solved in the current compound implementation in lttoolbox and lttoolbox-java is to have one symbol for word forms that may be a non-final part of a compund, compound-only-L, and one for forms that may end a compound, compound-R:

<pardef n="ep__n">
  <e>       <p><l>ane</l>       <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/><s n="compound-R"/></r></p></e>
  <e>       <p><l>ar</l>        <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  <e>       <p><l>en</l>        <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/><s n="compound-R"/></r></p></e>
  <e>       <p><l></l>          <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-R"/></r></p></e>
  <e r="LR"><p><l></l>          <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
</pardef>
...
<e lm="mobil"><i>mobil</i><par n="ep__n"/></e>
<e lm="telefon"><i>telefon</i><par n="ep__n"/></e>

The "only" in here compound-only-L means that this form may only appear as part of a compound, not alone. The reason for this is explained below.

Note: the compound symbols do not appear in the output, they only signal to lt-proc that this form may appear in compounds.

Limitations of lttoolbox compounding[edit]

The lttoolbox method is to first try analyzing a token without compounding, and only do compounding if the other methods would give an unknown word. Unknown words are made up of strings of characters from <alphabet>, separated by non-alphabetics or inconditional analyses.

So if the dictionary contains

<alphabet>abcdefghijklmnopqrstuvwxyz01-</alphabet>
…
<section id="main" type="standard">
  <e>       <p><l>oste</l>      <r>ost<s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
  <e>       <p><l>kaker</l>      <r>kake<s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  <e>       <p><l>100-</l>      <r>100<s n="det"/><s n="compound-only-L"/></r></p></e>
  <e>       <p><l>200-</l>      <r>200<s n="det"/><s n="compound-only-L"/></r></p></e>
</section>

Then you'll get compound analyses of "ostekake" and "100-kake" but not "200-kake" ("2" is missing from alphabet).

But if you have an inconditional analysis of numbers and dashes like

<section id="main" type="inconditional">
  <e>       <re>[0-9]+</re><p><l></l>      <r><s n="det"/><s n="qnt"/><s n="pl"/></r></p></e>
  <e>       <p><l>-</l>      <r>-<s n="guio"/></r></p></e>
</section>

then you won't even get "100-kake", it'll become

^100<det>$^-<guio>$^kake<n>$

Epenthetics[edit]

Many compounds have epenthetics, letters which connect the two words but perhaps only occur in compounds:

  • kransekake => krans+e+kake ≈ 'ring cake'
  • ungdomsfyll => ungdom+s+fyll = 'youth drunkenness'

The epenthetics do not have a meaning in themselves, and can typically only appear between compound parts (not when the word is by itself). (Their existance is purely phonological, "because nobody wants awkward morphemes"[1].)

Epenthetic analysis[edit]

We solve the analysis in lttoolbox and lttoolbox-java by the following type of paradigm:

<pardef n="krans__n">
  <e>       <p><l>ane</l>       <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/><s n="compound-R"/></r></p></e>
  <e>       <p><l>ar</l>        <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/><s n="compound-R"/></r></p></e>
  <e>       <p><l>en</l>        <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/><s n="compound-R"/></r></p></e>
  <e>       <p><l></l>          <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-R"/></r></p></e>
  <e r="LR"><p><l>e</l>         <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="compound-only-L"/></r></p></e>
</pardef>

As noted above, we have a difference between

  • compound-R meaning "this might be the right part of a compound, but might also stand alone", and
  • compound-only-L meaning "this might be the left part of a compound, but can't stand alone".

The above paradigm specifies that "kranse" may only appear as a left-part of a compound, it can't stand alone.

Objection 1: why not have just compound-L and compound-R, meaning "might be the left/right part or might stand alone"? Because you're never going to see "kakee" on its own, right?
Answer: True, we won't see "kakee" on its own, but "kranse" is a verb with another meaning, and tagging the noun as "compound-L" and allowing it to be analysed on its own would lead to unnecessary ambiguity.
Objection 2: Hey, you've got compound-only-L but not compound-L and not compund-only-R. You can't analyse Klingon!
Answer: This isn't even implemented yet. Once two symbols are implemented and working, we can think about adding the rest of the possibilites.
Objection 3: You really should have some sort of compound-both symbol for your <l/> entry instead of one with compound-only-L and one with compound-R, it'll mess up your tagger.
Answer: These symbols are removed in the output.

Epenthetic generation[edit]

When generating a compound, apertium-nn-nb at least simply adds the tag <cmp> to the left-part. The generator can then tell whether the left-part should have an epenthetic or not.

Example: after transfer we might have ^krans<n><m><sg><ind><cmp>$^kake<n><f><sg><ind>$. Then the generator just needs a pardef for krans that has the line

  <e r="RL"><p><l>e</l>         <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="cmp"/></r></p></e>

in order to output the correct kransekake (epenthetic e).

If the output from transfer is ^gulrot<n><m><sg><ind><cmp>$^kake<n><f><sg><ind>$ ("carrot cake"), and the generator has

  <e r="RL"><p><l></l>         <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/><s n="cmp"/></r></p></e>

in the pardef for gulrot, we get the correct gulrotkake (no epenthetic).

Meaning of epenthetics[edit]

The epenthetic has no meaning in itself, but if you can have the same word with or without an epenthetic, it might signal a difference. Compare:

  • sjefsekretær = 'chief of secretaries' (Norwegian)
  • sjefssekretær = 'the boss's secretary'
  • gjestfri = 'hospitable'
  • gjestefri = 'guest free' (as in "guest free zone")

(gjestfri would typically be listed in the dictionary, gjestefri being compositional.)

An epenthetic can also determine the meaning of an already ambiguous word-part:

  • tredørsvarianten = 'the three-door model'
  • tredørvarianten = 'the type of the wooden door'

Solution: add these to your dictionaries :-)

Outstanding questions[edit]

Stream symbol[edit]

We could use ~ instead of +, this would allow pretransfer to output '^$' instead of '^ $' for compounds (so transfer would no longer have to delete spaces)

This would need a small change in the tagger too. Can anyone think of a situation where it would need to be treated differently to normal def-mults?

Dix format[edit]

Currently we use

<s n="compound-only-L"/>
<s n="compound-R"/>

this could be changed to

<c r="L"/>
<c r="R"/>
We could also change "R" to only mark compound paths (so it would mean "compound-only-R"), since having a path that's both a compound part and usable as a full word is easily done with a pardef
Changes to dix format should be coordinated with unifying the metadix format

German/Capitalised compounds vs (proper noun) regexes[edit]

Some pairs use a regex to catch proper nouns, acronyms etc. Since regexes are no different from other entries in the FST, these will match before we get a chance to test for compounding. This means German nouns or capitalised compounds in other languages will never get a decompounding analysis.

When to compound[edit]

You shouldn't allow compounding on all words. In particular, if you allow compounding on all short words, you will get horrible results:

$ echo bildreportagen |apertium -d . swe-dan
billede #rids mugmide tagene

In Scandinavian dixen, it's common to have two versions of pardefs, one with and one without compounding:

<pardef n="ep__n">
  <e>       <p><l>ane</l>   <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p><par n="cp-R"/></e>
  <e>       <p><l>ar</l>    <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p><par n="cp-R"/></e>
  <e>       <p><l>en</l>    <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p><par n="cp-R"/></e>
  <e>       <p><l></l>      <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="cp-both\Ø_LR_s"/></e>
</pardef>
<pardef n="ep_no-cp__n">
  <e>       <p><l>ane</l>   <r><s n="n"/><s n="m"/><s n="pl"/><s n="def"/></r></p></e>
  <e>       <p><l>ar</l>    <r><s n="n"/><s n="m"/><s n="pl"/><s n="ind"/></r></p></e>
  <e>       <p><l>en</l>    <r><s n="n"/><s n="m"/><s n="sg"/><s n="def"/></r></p></e>
  <e>       <p><l></l>      <r><s n="n"/><s n="m"/><s n="sg"/><s n="ind"/></r></p><par n="no-cp\Ø"/></e>
</pardef>

(where cp-R adds compound-R, cp-both\Ø_LR_s adds compound-only-L with no epenthetic, LR with s epenthetic, and compound-R)

See also[edit]

Further reading[edit]