Apertium has moved from SourceForge to GitHub.
If you have any questions, please come and talk to us on #apertium on irc.freenode.net or contact the GitHub migration team.

Apertium stream format

From Apertium
Revision as of 17:27, 28 February 2019 by Sushain (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

En français


This page describes the stream format used in the Apertium machine translation platform.

[edit] Characters

[edit] Reserved

Reserved characters should only appear escaped in the input stream unless they are part of a lexical unit, chunk or superblank.

  • The characters ^ and $ are reserved for delimiting lexical units
  • The character / is reserved for delimiting analyses in ambiguous lexical units
  • The characters < and > are reserved for encapsulating tags
  • The characters { and } are reserved for delimiting chunks
  • The character \ is the escape character

[edit] Special

The following have special meaning at the start of an analysis:

  • Asterisk, '*' -- Unanalysed word.
  • At sign, '@' -- Untranslated lemma.
  • Hash sign, '#'
    • In morphological generation -- Unable to generate surface form from lexical unit (escape this to use # in lemmas)
    • In morphological analysis -- Start of invariable part of multiword marker (escape this to use # in lemmas)
  • Plus symbol, '+' -- Joined lexical units (escape this to use + in lemmas)
  • Tilde '~' -- Word needs treating by post-generator

[edit] Python parsing library

If you're writing a python script that needs to handle the Apertium stream format, try the excellent https://github.com/apertium/streamparser which lets you do

from streamparser import parse_file, mainpos, reading_to_string
for blank, lu in parse_file(file, with_text=True): 
    analyses = lu.readings
    firstreading = analyses[0]
    surfaceform = lu.wordform
    # rewrite to print only the first reading (and surface/word form):
    # convenience function to grab the first part of speech of the first reading:
    mainpos = mainpos(lu)

etc. without having to worry about superblanks and escaped characters and such :-)

Here's an example used in testvoc, this one splits ambiguous readings like ^foo/bar<n>/fie<ij>$ into ^foo/bar<n>$ ^foo/fie<ij>$, keeping the (super)blanks and newlines in between unchanged:

from streamparser import parse_file, reading_to_string
import sys
for blank, lu in parse_file(sys.stdin, with_text=True):
    print(blank+" ".join("^{}/{}$".format(lu.wordform, reading_to_string(r))
                         for r in lu.readings),

Here's a one-liner to print the lemmas of each word:

$ echo fisk bank kake|lt-proc nno-nob.automorf.bin|python3 -c  'import sys, streamparser; print ("\n".join("\t".join(set(s.baseform for r in lu.readings for s in r)) for lu in streamparser.parse_file(sys.stdin)))'

An alternative python lib: https://github.com/krvoje/apertium-transfer-dsl/blob/master/apertium/stream_entities.py https://github.com/krvoje/apertium-transfer-dsl/blob/master/apertium/stream_reader.py

[edit] Ruby parsing library

If you're writing a ruby script that needs to handle the Apertium stream format, you might want to try https://github.com/veer66/reinarb which seems similar to the Python streamparser

[edit] Formatted input

See also: Format handling

F = formatted text, T = text to be analysed.

Formatted text is treated as a single whitespace by all stages.

[<em>]this is[<\/em> ]a[ <b>]test.[][<\/b>]

|____|       |_______| |____|     |_______|
   |            |        |            |
   F            F        F            F
[<em>]this is[<\/em> ]a[ <b>]test.[][<\/b>]
      |______|        |      |____|
          |           |        | 
          T           T        T

[edit] Analyses

S = surface form, L = lemma.


   |    | |________|
   S    L    TAGS





                                                JOINED MORPHEMES

^take it away/take<vblex><sep><inf>+prpers<prn><obj><p3><nt><sg># away/take<vblex><sep><pres>+prpers<prn><obj><p3><nt><sg># away$

              |___|                                             |_____|
                |                                                   |
             LEMMA HEAD                                        LEMMA QUEUE

[edit] Chunks

See also: Chunking

^Verbcj<SV><vblex><ifi><p3><sg>{^come<vblex><ifi><p3><sg>$}$ ^pr<PREP>{^to<pr>$}$ ^det_nom<SN><f><sg>{^the<det><def><3>$ ^beach<n><3>$}$

   |   |______________________||__________________________|                                                          |
 CHUNK      CHUNK TAGS              LEXICAL UNITS IN                                                               LINKED
  NAME                                  THE CHUNK                                                                   TAG


^det_nom<SN><f><sg>{^the<det><def><3>$ ^beach<n><3>$}$

                                POINTERS TO CHUNK TAGS
        <1> <2> <3>     

[edit] See also

Personal tools