Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Inconditional section

From Apertium
(Difference between revisions)
Jump to: navigation, search
(Link to French page)
(inconditional)
Line 3: Line 3:
 
==inconditional==
 
==inconditional==
   
An inconditional ('unconditional') section of a dictionary typically contains punctuation, and such things.
+
An 'inconditional' section of a dictionary typically contains punctuation, and such things. The section type is used to change how tokenisation works.
   
The main section of a dictionary works on a longest-match basis.
+
In detail:
  +
Analysis in lttoolbox works in a left-to-right longest match fashion. We read characters from input, trying to match them in the transducer, and if we've reached a final transition when at the "end" of the input word, we can output the analysis (or we can try matching something even longer, but if not we use the match we found). But how do we know when we're at the end of an input word? Any "blank" character (not in <alphabet> in .dix files) is allowed to separate words, so e.g. spaces or other strange characters can separate words. But we may also say that certain words-with-analyses can separate words – to mark an analysis as "may act as word-separator", we put it in the 'inconditional' section.
   
Inconditional means 'if you see this, stop processing immediately and start reading a new word'. Stop when you reach the end of a possible transduction.
+
<!-- This is wrong:
  +
"Inconditional means 'if you see this, stop processing immediately and start reading a new word'. Stop when you reach the end of a possible transduction."
  +
We can have '.' in inconditional section and still get one long analysis of "Dr. Octagon" if "Dr. Octagon" is in e.g. the standard section. But if there was no analysis of the string up until the '.', then we may immediately output an unknown.
  +
-->
   
You could say that the "only" difference is that a space is not required to start a new match.
+
In summary, a space (or other blank) is not required to end the analysis of the preceding string, and neither is a blank required to start a new analysis afterwards.
   
 
<pre>
 
<pre>
Line 16: Line 16:
 
</pre>
 
</pre>
   
It doesn't need the space between 23 and men because numbers are in an 'inconditional' section.
+
In the above, we don't need the space between 23 and men because numbers are in an 'inconditional' section.
   
  +
  +
We can get some weird effects by putting plain characters in 'inconditional':
 
<pre>
 
<pre>
 
<dictionary>
 
<dictionary>

Revision as of 09:09, 8 June 2019

En français

Contents

inconditional

An 'inconditional' section of a dictionary typically contains punctuation, and such things. The section type is used to change how tokenisation works.

In detail: Analysis in lttoolbox works in a left-to-right longest match fashion. We read characters from input, trying to match them in the transducer, and if we've reached a final transition when at the "end" of the input word, we can output the analysis (or we can try matching something even longer, but if not we use the match we found). But how do we know when we're at the end of an input word? Any "blank" character (not in <alphabet> in .dix files) is allowed to separate words, so e.g. spaces or other strange characters can separate words. But we may also say that certain words-with-analyses can separate words – to mark an analysis as "may act as word-separator", we put it in the 'inconditional' section.


In summary, a space (or other blank) is not required to end the analysis of the preceding string, and neither is a blank required to start a new analysis afterwards.

$ echo 23men |apertium -d . en-it-anmor
^23/23<num>$^men/man<n><pl>$^./.<sent>$

In the above, we don't need the space between 23 and men because numbers are in an 'inconditional' section.


We can get some weird effects by putting plain characters in 'inconditional':

<dictionary>
  <alphabet>ab</alphabet>
  <sdefs>
    <sdef n="aa"/>
    <sdef n="ab"/>
  </sdefs>
  <section id="foo" type="inconditional">
    <e><p><l>a</l><r>a<s n="aa"/></r></p></e>
    <e><p><l>aa</l><r>aa<s n="aa"/></r></p></e>
  </section>
</dictionary>

$ echo aaa |lt-proc  sample.bin
^aa/aa<aa>$^a/a<aa>$

$ echo aaaa |lt-proc  sample.bin
^aa/aa<aa>$^aa/aa<aa>$

$ echo aaaaa |lt-proc  sample.bin
^aa/aa<aa>$^aa/aa<aa>$^a/a<aa>$

postblank / preblank

The postblank and preblank sections work exactly like inconditional with respect to how they tokenise the input. The only difference is that anything in a postblank section will make lt-proc output a space after the token (in preblank, before the token).

So if "☃" is in postblank (tagged sent), and "foo" and "bar" are in a regular section (tagged n), then we get:

$ echo 'foo☃bar' | lt-proc analyser.bin
^foo/foo<n>$^☃/☃<sent>$ ^bar/bar<n>$

If "☃" were in preblank, we'd get:

$ echo 'foo☃bar' | lt-proc analyser.bin
^foo/foo<n>$ ^☃/☃<sent>$^bar/bar<n>$

Why is this useful?

TODO

See also

Personal tools