Difference between revisions of "Talk:Automatically trimming a monodix"
Jump to navigation
Jump to search
| (36 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| + | == HFST: possible to overcome compound overgeneration? == |
||
| − | ==Implementing automatic trimming in lttoolbox== |
||
| − | The simplest method seems to be to first create the analyser in the normal way, then loop through all its states (see transducer.cc:Transducer::closure for a loop example), trying to do the same steps in parallel with the compiled bidix: |
||
| + | Assuming we ''don't use flags'', we can compile an HFST transducer through [[ATT]] format to a working lttoolbox transducer. |
||
| − | <pre> |
||
| − | trim(current_a, current_b): |
||
| + | Now the issue is that compounds are plain transitions back into some lexicon, without the compound-only-L/compound-R tags, so even though lt-trim should trim the compounds correctly (by treating them like <j/> transitions), the resulting analyser will over-generate: |
||
| − | for symbol, next_a in analyser.transitions[current_a]: |
||
| + | <pre>jīvitarēkha/jīvitarēkha<n><sg><nom>/jīvitaṁ<n><cmp>+rēkha<n><sg><nom></pre> |
||
| − | found = false |
||
| + | Ie. we will get non-lexicalised compound analyses along with the lexicalised ones. |
||
| − | for s, next_b in bidix.transitions[current_b]: |
||
| − | if s==symbol: |
||
| − | trim(next_a, next_b) |
||
| − | found = true |
||
| − | if seen tags: |
||
| − | found = true |
||
| + | One way to overcome this is to first compile the analyser from the ATT, then: |
||
| − | if !found && !current_b.isFinal(): |
||
| + | * go through it building a new version, but on seeing a +, we: |
||
| − | delete symbol from analyser.transitions[current_a] |
||
| + | ** replace the transition with a single compound-only-L tag transitioning into final state |
||
| + | ** let partial=copy_until_final(the_plus_transition) and make a transition from start into partial, and we connect the final state of partial with a single tag compound-R into the final state of our new FST |
||
| + | If the function copy_until_final sees a +, that transition is discarded, but a compound-only-L tag is added. |
||
| − | // else: all transitions from this point on will just be carried over unchanged by bidix |
||
| − | trim(analyser.initial, bidix.initial) |
||
| − | </pre> |
||
| + | Note: this compounding method won't let you do stuff like "only allow adj adj or noun noun compounds" like you can in HFST. |
||
| − | [[Image:Product-automaton-intersection.png|thumb|400px|right|product automaton for intersection]] |
||
| + | ::You can do that in the morphotactics though (e.g. adjective paradigms only redirect to the adjectivestem lexicon.) - [[User:Francis Tyers|Francis Tyers]] ([[User talk:Francis Tyers|talk]]) 13:33, 22 May 2014 (CEST) |
||
| − | Trimming while reading the XML file might have lower memory usage, but seems like more work, since pardefs are read before we get to an "initial" state. |
||
| + | : Uh, what happens if you simply specify compound-R and compound-only-L tags in the lexc? |
||
| − | A slightly different approach is to create the '''product automaton''' for intersection, marking as final only state-pairs where both parts of the state-pair are final in the original automata. Minimisation should remove unreachable state-pairs. However, this quickly blows up in memory usage since it creates ''all'' possible state pairs first (cartesian product), not just the useful ones. |
||
| + | ::This is also an option, while we were at it I'd choose a prettier couple of symbols though ;) - [[User:Francis Tyers|Francis Tyers]] ([[User talk:Francis Tyers|talk]]) 13:33, 22 May 2014 (CEST) |
||
| − | https://github.com/unhammer/lttoolbox/branches has some experiments, see e.g. branches product-intersection and df-intersection |
||
| − | <br clear="all" /> |
||
| + | Say we have the FST "bidix", can we create "<code>bidix [^+]* (+ bidix [^+]*)*</code>" (where + is just the literal join symbol) and trim with that? |
||
| − | ==Worse is better== |
||
| + | : seems to work! |
||
| − | If it's difficult to get a complete lt-trim solution that handles <g> perfectly a stop-gap might be to distribute a dictionary of the multiwords with the language pair, and leave only "simple" words in the monolingual package dictionary. The multiwords would be manually trimmed to the pair, the simple words trimmed with lt-trim, and a new <code>lt-merge</code> command would merge the two compiled dictionaries (as seperate sections). |
||
| − | |||
| − | (An lt-merge command might also be helpful when compiling att transducers, which makes anything that looks like punctuation [[inconditional]], and anything else standard.) |
||
| − | |||
| − | ==#-type multiwords== |
||
| − | An idea for implementation: first "preprocess" the bidix so it has the same format as the analyser, ie. instead of "take# out<vblex>" it has "take<vblex># out". |
||
| − | |||
| − | Say you have these entries in the bidix: <pre> |
||
| − | take# out<vblex> |
||
| − | take# out<n> |
||
| − | take# in<vblex> |
||
| − | take<n> |
||
| − | take<vblex> |
||
| − | </pre> |
||
| − | |||
| − | You do a depth-first traversal, and then after reading all of "take" you see t=transition(A, B, ε, #). You remove that transition, and instead add the result of unpretransfer(t), which returns an ε-transition into a new transducer that represents these partial analyses: |
||
| − | <pre><vblex># out |
||
| − | <n># out |
||
| − | <vblex># in</pre> |
||
| − | |||
| − | ===pseudocode== |
||
| − | <pre> |
||
| − | seen = set() |
||
| − | todo = [initial] |
||
| − | while todo: |
||
| − | src = todo.pop() |
||
| − | for left, right, trg in transitions[src]: |
||
| − | if left == '#': |
||
| − | transitions[src].remove('#') |
||
| − | transitions[src][epsilon] = unpretransfer(trg) |
||
| − | else: |
||
| − | if trg not in seen: |
||
| − | todo.push(trg) |
||
| − | seen.insert(trg) |
||
| − | |||
| − | def unpretransfer(start): |
||
| − | lemq = Transducer() |
||
| − | todo = [ ( start, lemq, lemq.initial ) ] |
||
| − | # Each searchstate in the stack is a transition in this FST, |
||
| − | # along with a lemq-FST which is a completely linear transducer |
||
| − | # (ie. all states have at most one transition), |
||
| − | # as well as a pointer to the currently "last" state in the lemq. |
||
| − | while todo: |
||
| − | src, lemq, lemq_last = todo.pop() |
||
| − | for left, right, trg in self.transitions[src]: |
||
| − | if not self.isTag(left): |
||
| − | # We're reading a lemq; append it to the current one: |
||
| − | lemq_trg = lemq.newState() |
||
| − | lemq.linkStates(lemq_last, left, right, lemq_trg) |
||
| − | todo.push(( trg, lemq, lemq_trg )) |
||
| − | else: |
||
| − | # We've seen the first tag! Put it at the start: |
||
| − | self.linkStates(start, left, right, trg) |
||
| − | lemq.finals.insert(lemq_last) |
||
| − | todo.push(( trg, lemq, lemq_last )) |
||
| − | if trg in self.finals: |
||
| − | unmarkfinals.insert(trg) |
||
| − | self.insertTransducer(trg, lemq) |
||
| − | # TODO: how does this interact with appendDotStar? |
||
| − | </pre> |
||
| − | |||
| − | == Compounds vs trimming in sme == |
||
| − | |||
| − | The sme.lexc can't be trimmed using the simple HFST trick, due to compounds. |
||
| − | |||
| − | Say you have '''cake n sg''', '''cake n pl''', '''beer n pl''' and '''beer n sg''' in monodix, while bidix has '''beer n''' and '''wine n'''. The HFST method without compounding is to intersect '''(cake|beer) n (sg|pl)''' with '''(beer|wine) n .*''' to get '''beer n (sg|pl)'''. |
||
| − | |||
| − | But HFST represents compounding as a transition from the end of the singular noun to the beginning of the (noun) transducer, so a compounding HFST actually looks like |
||
| − | : '''((cake|beer) n sg)*(cake|beer) n (sg|pl)''' |
||
| − | The intersection of this with |
||
| − | : '''(beer|wine) n .*''' |
||
| − | is |
||
| − | : '''(beer n sg)*(cake|beer) n (sg|pl) | beer n pl''' |
||
| − | when it should have been |
||
| − | : '''(beer n sg)*(beer n (sg|pl)''' |
||
| − | |||
| − | |||
| − | Lttoolbox doesn't represent compounding by extra circular transitions, but instead by a special restart symbol interpreted while analysing. |
||
| − | When we have lt-trim we will be able to make it understand compounds by e.g. restarting on + |
||
Latest revision as of 14:47, 20 June 2014
HFST: possible to overcome compound overgeneration?[edit]
Assuming we don't use flags, we can compile an HFST transducer through ATT format to a working lttoolbox transducer.
Now the issue is that compounds are plain transitions back into some lexicon, without the compound-only-L/compound-R tags, so even though lt-trim should trim the compounds correctly (by treating them like <j/> transitions), the resulting analyser will over-generate:
jīvitarēkha/jīvitarēkha<n><sg><nom>/jīvitaṁ<n><cmp>+rēkha<n><sg><nom>
Ie. we will get non-lexicalised compound analyses along with the lexicalised ones.
One way to overcome this is to first compile the analyser from the ATT, then:
- go through it building a new version, but on seeing a +, we:
- replace the transition with a single compound-only-L tag transitioning into final state
- let partial=copy_until_final(the_plus_transition) and make a transition from start into partial, and we connect the final state of partial with a single tag compound-R into the final state of our new FST
If the function copy_until_final sees a +, that transition is discarded, but a compound-only-L tag is added.
Note: this compounding method won't let you do stuff like "only allow adj adj or noun noun compounds" like you can in HFST.
- You can do that in the morphotactics though (e.g. adjective paradigms only redirect to the adjectivestem lexicon.) - Francis Tyers (talk) 13:33, 22 May 2014 (CEST)
- Uh, what happens if you simply specify compound-R and compound-only-L tags in the lexc?
- This is also an option, while we were at it I'd choose a prettier couple of symbols though ;) - Francis Tyers (talk) 13:33, 22 May 2014 (CEST)
Say we have the FST "bidix", can we create "bidix [^+]* (+ bidix [^+]*)*" (where + is just the literal join symbol) and trim with that?
- seems to work!
