Difference between revisions of "Talk:Automatically trimming a monodix"
| Line 39: | Line 39: | ||
* more tests? |
* more tests? |
||
* if no objections: git svn dcommit |
* if no objections: git svn dcommit |
||
| − | |||
| − | == + type multiwords == |
||
| − | join (<j>) is handled by setting the next trimmer (bidix) target state to be trimmer.initial, so we simply skip to the start. |
||
| − | === + and # together === |
||
| − | ''However'', if we have a + followed by a # in monodix, we need to get back to where we were after reading the + part. An example: |
||
| − | <pre>monodix, one entry: |
||
| − | foo<n><ind>+bar<pr># fie |
||
| − | |||
| − | bidix, two entries: |
||
| − | foo# fie<n> |
||
| − | bar<pr> |
||
| − | |||
| − | preprocessed (prefixed+lemq-moved) bidix, two entries: |
||
| − | foo<n>[<n>|<ind>|<pr>]*# fie |
||
| − | bar<pr>[<n>|<ind>|<pr>]* |
||
| − | </pre> |
||
| − | |||
| − | We start by matching on these paths: <pre> |
||
| − | monodix string foo<n><ind> + bar<pr> # fie |
||
| − | monodix states g i j k |
||
| − | bidix string foo<n>[<n>|<ind>|<pr>]* # fie |
||
| − | bidix states l m |
||
| − | </pre> |
||
| − | and get as far as state pair (i,m), having consumed <pre>foo<n><ind></pre> before seeing a +. |
||
| − | |||
| − | Now, the next SearchState records both the next monodix state (j, the state after +), the next bidix state (l, the bidix initial state), and the bidix state we were in when we saw the plus (m, at the end of the bidix tag list). So in the code SearchState next = (this_trg, trimmer.initial, trimmer_src). |
||
| − | |||
| − | At (i,l,m) we've consumed <pre>foo<n><ind></pre> and have <pre>bar<pr># fie</pre> left, and can read from bidix initial l:<pre> |
||
| − | monodix string bar<pr> # fie |
||
| − | monodix states j k |
||
| − | bidix string bar<n>[<n>|<ind>|<pr>]* |
||
| − | bidix states l n |
||
| − | </pre> |
||
| − | |||
| − | We match all of <pre>bar<pr></pre> and then see a # in monodix, giving SearchState (k,n,m). Since both the third part of SearchState is set to state m (where m!=n), and we've seen a #, we jump back to that state m, from where we can match <code># fie</code>, and we're done. |
||
== Compounds vs trimming in HFST == |
== Compounds vs trimming in HFST == |
||
Revision as of 08:32, 11 February 2014
Implementing automatic trimming in lttoolbox
The current method: compile the analyser in the normal way, then lt-trim loads it and loops through all its states, trying to do the same steps in parallel with the compiled bidix:
trim(current_a, current_b):
for symbol, next_a in analyser.transitions[current_a]:
found = false
for s, next_b in bidix.transitions[current_b]:
if s==symbol:
trim(next_a, next_b)
found = true
if seen tags:
found = true
if !found && !current_b.isFinal():
delete symbol from analyser.transitions[current_a]
// else: all transitions from this point on will just be carried over unchanged by bidix
trim(analyser.initial, bidix.initial)
Trimming while reading the XML file might have lower memory usage, but seems like more work, since pardefs are read before we get to an "initial" state.
A slightly different approach is to create the product automaton for intersection, marking as final only state-pairs where both parts of the state-pair are final in the original automata. Minimisation should remove unreachable state-pairs. However, this quickly blows up in memory usage since it creates all possible state pairs first (cartesian product), not just the useful ones.
https://github.com/unhammer/lttoolbox/branches has some experiments, see e.g. branches product-intersection and df-intersection
TODO
- Would it make things faster if we first clear out the right-sides of the bidix and minimize that?
- remove all those ifdef DEBUG's and wcerr's
- more tests?
- if no objections: git svn dcommit
Compounds vs trimming in HFST
The sme.lexc can't be trimmed using the simple HFST trick, due to compounds.
Say you have cake n sg, cake n pl, beer n pl and beer n sg in monodix, while bidix has beer n and wine n. The HFST method without compounding is to intersect (cake|beer) n (sg|pl) with (beer|wine) n .* to get beer n (sg|pl).
But HFST represents compounding as a transition from the end of the singular noun to the beginning of the (noun) transducer, so a compounding HFST actually looks like
- ((cake|beer) n sg)*(cake|beer) n (sg|pl)
The intersection of this with
- (beer|wine) n .*
is
- (beer n sg)*(cake|beer) n (sg|pl) | beer n pl
when it should have been
- (beer n sg)*(beer n (sg|pl)
Lttoolbox doesn't represent compounding by extra circular transitions, but instead by a special restart symbol interpreted while analysing.
lt-trim is able to understand compounds by simply skipping the compund tags
