Apertium stream format - Revision history

Veer66: cl-apertium-stream

2022-03-29T07:12:49Z

cl-apertium-stream

2021-04-05T22:54:42Z

Special: link to longer explanation of +

2019-02-28T16:27:47Z

fix pep8 changes

2018-01-22T12:30:07Z

Characters

2018-01-22T12:27:27Z

Special

2017-02-06T08:22:13Z

Python parsing library

2015-12-28T10:21:06Z

Python parsing library

2015-12-28T10:17:19Z

Python parsing library

2015-04-15T10:18:47Z

Python parsing library

2015-01-30T10:03:48Z

copypasta

@@ Line 65: / Line 65: @@
 An alternative python lib: https://github.com/krvoje/apertium-transfer-dsl/blob/master/apertium/stream_entities.py https://github.com/krvoje/apertium-transfer-dsl/blob/master/apertium/stream_reader.py
-==Ruby parsing library==
+==Common Lisp parsing library==
+cl-apertium-stream[https://github.com/veer66/cl-apertium-stream] is a library written in Common Lisp for parsing Apertium stream and generating Apertium stream from parsed data. It is developed based on the discontinued Ruby library[https://github.com/veer66/reinarb]. cl-apertium-stream is data-driven. Its parsed data is a list, keyword, and string combination without any new type/class. So further processing is based on ordinary list operations. cl-apertium-stream handles Apertium stream format by declarative Esrap[https://github.com/scymtym/esrap] rules.
-If you're writing a ruby script that needs to handle the Apertium stream format, you might want to try https://github.com/veer66/reinarb which seems similar to the Python streamparser
 ==Formatted input==

@@ Line 25: / Line 25: @@
 ** In morphological generation -- Unable to generate [[surface form]] from [[lexical unit]] (escape this to use # in lemmas)
 ** In morphological analysis -- Start of invariable part of multiword marker  (escape this to use # in lemmas)
-* Plus symbol, '<code><nowiki>+</nowiki></code>' -- Joined lexical units (escape this to use + in lemmas)
+* Plus symbol, '<code><nowiki>+</nowiki></code>' -- [[Conjoined lexical units|Joined lexical units]] (escape this to use + in lemmas)
 * Tilde '<code><nowiki>~</nowiki></code>' -- Word needs treating by [[post-generator]]

@@ Line 33: / Line 33: @@
 <pre>
-from streamparser import parse_file, mainpos, readingToString
+from streamparser import parse_file, mainpos, reading_to_string
-for blank, lu in parse_file(file, withText=True):
+for blank, lu in parse_file(file, with_text=True):
     analyses = lu.readings
     firstreading = analyses[0]
@@ Line 40: / Line 40: @@
     # rewrite to print only the first reading (and surface/word form):
     print("^{}/{}$".format(surfaceform,
-                           readingToString(firstreading)))
+                           reading_to_string(firstreading)))
     # convenience function to grab the first part of speech of the first reading:
     mainpos = mainpos(lu)
@@ Line 49: / Line 49: @@
 Here's an example used in testvoc, this one splits ambiguous readings like <code><nowiki>^foo/bar<n>/fie<ij>$</nowiki></code> into  <code><nowiki>^foo/bar<n>$ ^foo/fie<ij>$</nowiki></code>, keeping the (super)blanks and newlines in between unchanged:
 <pre>
-from streamparser import parse_file, readingToString
+from streamparser import parse_file, reading_to_string
 import sys
-for blank, lu in parse_file(sys.stdin, withText=True):
+for blank, lu in parse_file(sys.stdin, with_text=True):
-    print(blank+" ".join("^{}/{}$".format(lu.wordform, readingToString(r))
+    print(blank+" ".join("^{}/{}$".format(lu.wordform, reading_to_string(r))
                          for r in lu.readings),
           end="")

@@ Line 23: / Line 23: @@
 * At sign, '<code><nowiki>@</nowiki></code>' -- Untranslated [[lemma]].
 * Hash sign, '<code><nowiki>#</nowiki></code>'
-** In morphological generation -- Unable to generate [[surface form]] from [[lexical unit]].
+** In morphological generation -- Unable to generate [[surface form]] from [[lexical unit]] (escape this to use # in lemmas)
-** In morphological analysis -- Start of invariable part of multiword marker.
+** In morphological analysis -- Start of invariable part of multiword marker  (escape this to use # in lemmas)
-* Tilde '<code><nowiki>~</nowiki></code>' -- Word needs treating by [[post-generator]].
+* Plus symbol, '<code><nowiki>+</nowiki></code>' -- Joined lexical units (escape this to use + in lemmas)
+* Tilde '<code><nowiki>~</nowiki></code>' -- Word needs treating by [[post-generator]]
 ==Python parsing library==

@@ Line 17: / Line 17: @@
 ===Special===
+The following have special meaning at the start of an analysis:
 * Asterisk, '<code><nowiki>*</nowiki></code>' -- Unanalysed word.
@@ Line 23: / Line 25: @@
 ** In morphological generation -- Unable to generate [[surface form]] from [[lexical unit]].
 ** In morphological analysis -- Start of invariable part of multiword marker.
-* Plus symbol, '<code><nowiki>+</nowiki></code>' -- Joined lexical units
 * Tilde '<code><nowiki>~</nowiki></code>' -- Word needs treating by [[post-generator]].

@@ Line 32: / Line 32: @@
 <pre>
 from streamparser import parse_file, mainpos, readingToString
-for blank, lu in parse_file(file):
+for blank, lu in parse_file(file, withText=True):
     analyses = lu.readings
     firstreading = analyses[0]
@@ Line 55: / Line 55: @@
 </pre>
+Here's a one-liner to print the lemmas of each word:
+<pre>
+$ echo fisk bank kake|lt-proc nno-nob.automorf.bin|python3 -c  'import sys, streamparser; print ("\n".join("\t".join(set(s.baseform for r in lu.readings for s in r)) for lu in streamparser.parse_file(sys.stdin)))'
+</pre>