Apertium separable - Revision history

Firespeaker: /* Creating the lsx-dictionary */

2025-05-05T15:24:52Z

Creating the lsx-dictionary

Unhammer: /* Matching forms */

2024-05-02T18:54:55Z

Matching forms

Unhammer: /* Troubleshooting */

2024-05-02T13:48:38Z

Troubleshooting

Popcorndude: j → d and delete some sections about bugs that don't apply anymore

2022-11-11T20:03:15Z

j → d and delete some sections about bugs that don't apply anymore

Show changes

Firespeaker: /* Installing */

2018-11-10T17:54:26Z

Installing

Bech: /* Example usages */ folding long lines of code

2018-05-15T11:58:43Z

Example usages: folding long lines of code

Shardulc: GitHub migration

2018-03-10T01:57:22Z

GitHub migration

Shardulc: /* See also */

2018-03-10T01:55:15Z

Ilnar.salimzyan: s/sourceforge link/github link/

2018-03-09T18:13:14Z

s/sourceforge link/github link/

Irene: /* The one-to-many bug */ oops formatting

2017-12-21T19:28:52Z

The one-to-many bug: oops formatting

@@ Line 61: / Line 61: @@
 </pre>
-Now we need to add the paradigm definitions, abbreviated to pardefs. These represent patterns of word orders. The following example represents words tagged as adjective, noun, noun phrase, and frequency adjectives. See the note below about the tags {{tag|w/}}, {{tag|t/}}, {{tag|j/}}. The lemma can be represented as anychars ({{tag|w/}}, such as in adj and n below; or by typing out the word itself, such as in freq-adv below. Pardefs can be used to create other pardefs, such as in SN below. Adding paradigms into the dictionary, we get:
+Now we need to add the paradigm definitions, abbreviated to pardefs. These represent patterns of word orders. The following example represents words tagged as adjective, noun, noun phrase, and frequency adjectives. See the note below about the tags {{tag|w/}}, {{tag|t/}}, {{tag|d/}}, and {{tag|j/}}. The lemma can be represented as anychars ({{tag|w/}}, such as in adj and n below; or by typing out the word itself, such as in freq-adv below. Pardefs can be used to create other pardefs, such as in SN below. Adding paradigms into the dictionary, we get:
 <pre>
 <dictionary type="separable">
@@ Line 139: / Line 139: @@
 * {{tag|t/}} stands for one or more tags (multicharacter symbols).
 * {{tag|d/}} stands for the word boundary symbol $
+* {{tag|j/}} stands for the word separator symbol +
 i.e.

@@ Line 279: / Line 279: @@
 </pre>
+'''NB''': If you are using [[HFST]] to create your lsx binary, you will need to run <code>lt-comp</code> with the <code>-S</code> option on your [[ATT]] file, e.g.
+<code>lt-comp -S lr sep.att sep.bin</code>
 ==Troubleshooting==

@@ Line 185: / Line 185: @@
 ^the<det><def><sp>$ ^Aragonese<n><sg>$ ^take# out<vblex><sep><past>$ ^Ramiro<np><ant><m><sg>$ ^of<pr>$ ^a<det><ind><sg>$
 ^monastery<n><sg>$ ^and<cnjcoo>$ ^make<vblex><pp>$ ^prpers<prn><obj><p3><m><sg>$ ^king<n><sg>$^.<sent>$
+</pre>
+==Matching forms==
+You can also use lsx-proc on readings that include forms. Run <code>apertium-tagger</code> with <code>-p</code> to ensure forms are not stripped off, and run <code>lsx-proc</code> with <code>-p</code> to enable analysing forms. Use <code>&lt;f/&gt;</code> to match the <code>/</code> between form and reading:
+<pre>
+$ cat sep.lsx
+<?xml version="1.0" encoding="UTF-8"?>
+<dictionary type="separable">
+  <alphabet></alphabet>
+  <sdefs>
+    <sdef n="np"/>
+    <sdef n="pr"/>
+    <sdef n="vblex"/>
+    <sdef n="adv"/>
+  </sdefs>
+  <pardefs>
+    <pardef n="reading" c="match and keep readings (incl. tagless/unknown). Includes end delimiter">
+      <e>   <i><f/><w/><d/></i>            </e>
+      <e>   <i><f/><w/><t/><d/></i>        </e>
+    </pardef>
+    <pardef n="reading:" c="match and drop readings (incl. tagless/unknown). Includes end delimiter">
+      <e><p><l><f/><w/><d/></l>    <r/></p></e>
+      <e><p><l><f/><w/><t/><d/></l><r/></p></e>
+    </pardef>
+    <pardef n="pr|jf" c="includes end delimiter">
+      <e><i><w/><f/><w/><s n="pr"/><t/><d/></i></e>
+      <e><i>jf.</i>     <par n="reading"/></e>
+    </pardef>
+  </pardefs>
+  <section id="main" type="standard">
+    <e c="merge">
+      <par n="pr|jf"/>
+      <p><l>lov</l> <r></r></p> <par n="reading:"/>
+      <p><l>om</l> <r></r></p> <par n="reading:"/>
+      <p><l>kake</l> <r></r></p> <par n="reading:"/>
+      <p><l></l> <r>lov<b/>om<b/>kake<s n="np"/><d/></r></p>
+    </e>
+    <e c="split">
+      <p><l>wouldnae</l> <r></r></p> <par n="reading:"/>
+      <p><l></l> <r>would<f/>will<s n="vblex"/><d space="no"/></r></p>
+      <p><l></l> <r>nae<f/>not<s n="adv"/><d/></r></p>
+    </e>
+    <!--
+        The "reading" pardefs above will match all readings (even
+        unknowns), so they really just filter on form.
+        But we could easily match on lemmas/tags as well, e.g the below entry would turn
+        ^aint/benot<vblex><adv>$
+        into
+        ^ai/be<vblex>$ ^nt/not<adv>$
+        but would not match
+        ^aint/havenot<vblex><adv>$
+    -->
+    <e c="require certain lemma/tags">
+      <p><l>aint<f/>benot<s n="vblex"/><s n="adv"/><d/></l> <r></r></p>
+      <p><l></l> <r>ai<f/>be<s n="vblex"/><d space="no"/></r></p>
+      <p><l></l> <r>nt<f/>not<s n="adv"/><d/></r></p>
+    </e>
+  </section>
+</dictionary>
+$ lsx-comp lr sep.lsx sep.bin
+main@standard 99 118
+$ echo '^jf./jf.<pr>$ ^lov/lov<n><m>$ ^om/om<pr>$ ^kake/kake<n><f>$' | lsx-proc -w -p sep.bin
+^jf./jf.<pr>$ ^lov/lov om kake<np>$
+$ echo '^wouldnae/willnot<vblex><adv>$' | lsx-proc -w -p sep.bin
+^would/will<vblex>$^nae/not<adv>$
+$ echo '^wouldnae/*wouldnae$' | lsx-proc -w -p sep.bin
+^would/will<vblex>$^nae/not<adv>$
+$ echo '^aint/benot<vblex><adv>$' | lsx-proc -w -p sep.bin
+^ai/be<vblex>$^nt/not<adv>$
+$ echo '^aint/havenot<vblex><adv>$' | lsx-proc -w -p sep.bin
+^aint/havenot<vblex><adv>$
 </pre>

@@ Line 4: / Line 4: @@
 ==Installing==
-Prerequisites and compilation are the same as lttoolbox and apertium. See [[Installation]]. On Debian/Ubuntu derivatives, it is part of the nightly repo as <code>apt-get install apertium-separable</code>.
+The module is part of the [[nightly repositories]] as <code>apt-get install apertium-separable</code>.
+If you'd like to compile it manually—e.g., for development purposes—you can follow these instructions:
-The code can be found at https://github.com/apertium/apertium-separable and instructions for compiling the module are:
+Prerequisites and compilation are the same as lttoolbox and apertium. See [[Installation]].
+The code can be found at [https://github.com/apertium/apertium-separable https://github.com/apertium/apertium-separable], and instructions for compiling the module are:
 <pre>
@@ Line 15: / Line 19: @@
 </pre>
-You'll need lttoolbox from git (or, greater than the current release 3.3.3) and associated libraries, and zlib (debian: zlib1g-dev).
+You'll need <code>lttoolbox</code> from git (or, greater than the current release 3.3.3) and associated libraries, and <code>zlib</code> (debian: <code>zlib1g-dev</code>).
-<s>It is not currently part of distributed Apertium binaries for other distros/OSs.</s>  It is now available via the nightly repositories as the <code>apertium-separable</code> module.
 ==Lexical transfer in the pipeline==

@@ Line 175: / Line 175: @@
 This is the output of feeding the sentence through <code> apertium-tagger </code> and then <code> apertium-pretransfer </code>:
 <pre>
-^the<det><def><sp>$ ^Aragonese<n><sg>$ ^take<vblex><past>$ ^Ramiro<np><ant><m><sg>$ ^out of<pr>$ ^a<det><ind><sg>$ ^monastery<n><sg>$ ^and<cnjcoo>$ ^make<vblex><pp>$ ^prpers<prn><obj><p3><m><sg>$ ^king<n><sg>$^.<sent>$
+^the<det><def><sp>$ ^Aragonese<n><sg>$ ^take<vblex><past>$ ^Ramiro<np><ant><m><sg>$ ^out of<pr>$ ^a<det><ind><sg>$
+^monastery<n><sg>$ ^and<cnjcoo>$ ^make<vblex><pp>$ ^prpers<prn><obj><p3><m><sg>$ ^king<n><sg>$^.<sent>$
 </pre>
 This is the output of feeding the output above through <code> lsx-proc </code> with apertium-eng-spa.eng-spa.lsx:
 <pre>
-^the<det><def><sp>$ ^Aragonese<n><sg>$ ^take# out<vblex><sep><past>$ ^Ramiro<np><ant><m><sg>$ ^of<pr>$ ^a<det><ind><sg>$ ^monastery<n><sg>$ ^and<cnjcoo>$ ^make<vblex><pp>$ ^prpers<prn><obj><p3><m><sg>$ ^king<n><sg>$^.<sent>$
+^the<det><def><sp>$ ^Aragonese<n><sg>$ ^take# out<vblex><sep><past>$ ^Ramiro<np><ant><m><sg>$ ^of<pr>$ ^a<det><ind><sg>$
+^monastery<n><sg>$ ^and<cnjcoo>$ ^make<vblex><pp>$ ^prpers<prn><obj><p3><m><sg>$ ^king<n><sg>$^.<sent>$
 </pre>

@@ Line 144: / Line 144: @@
 ** ^tall<adj>$
-A larger example dictionary can be found at https://svn.code.sf.net/p/apertium/svn/branches/apertium-separable/examples/apertium-eng-spa.eng-spa.lsx
+A larger example dictionary can be found at https://github.com/apertium/apertium-separable/blob/master/examples/apertium-eng-spa.eng-spa.lsx.
 The lsx dictionary file names are of the form <code> apertium-A-B.A-B.lsx </code>, where apertium-A-B is the name of the language pair. For example, file <code>apertium-eng-cat.eng-cat.lsx</code> is the lsx dictionary for the <code> eng-cat </code> pair. The names of the compiled binaries are of the form <code> apertium-A-B.autoseq.bin </code>. For example, <code> eng-cat.autoseq.bin </code>.
@@ Line 194: / Line 194: @@
 ===Complaints about step_override()===
-svn update in lttoolbox (and do make, make install) <br/>
+git pull in lttoolbox (and do make, make install) <br/>
 You'll need an up-to-date version of lttoolbox and associated libraries, and zlib (debian: zlib1g-dev). <br/>

@@ Line 327: / Line 327: @@
 ==See also==
-* https://svn.code.sf.net/p/apertium/svn/branches/apertium-separable
 * [[Apertium system architecture]]
 * GSOC project [[User:Irene/proposal | proposal]], [[User:Irene/workplan | workplan]], [[Lsx_module/report | report]]

@@ Line 305: / Line 305: @@
 we expect lr compilation to give the following behaviour:
+<pre>
 $ echo "^хабар ет<v><iv><ifi><p1><sg>$" | lsx-proc kaz-kir.autoseq.bin
-^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$ (changes)
+^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$
+</pre>
 and
+<pre>
 $ echo "^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$" | lsx-proc kaz-kir.autoseq.bin
-^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$ (does nothing)
+^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$
+</pre>
 WHEREAS with rl compilation (outputting with name revautoseq), we expect the following behaviour:
+<pre>
 $ echo "^хабар<n><nom>$ ^ет<v><iv><ifi><p1><sg>$" | lsx-proc kaz-kir.revautoseq.bin
-^хабар ет<v><iv><ifi><p1><sg>$ (changes)
+^хабар ет<v><iv><ifi><p1><sg>$
+</pre>
 and
+<pre>
 $ echo "^хабар ет<v><iv><ifi><p1><sg>$" | lsx-proc kaz-kir.revautoseq.bin
-^хабар ет<v><iv><ifi><p1><sg>$ (does nothing)
+^хабар ет<v><iv><ifi><p1><sg>$
+</pre>
 ==See also==