User:Firespeaker/HFST bug

In 2011, a bug in how HFST handles words containing spaces was documented and resolved, but it introduced a new bug. This page documents the new behaviour.

text.lexc

Multichar_Symbols

% 

LEXICON Root

erke:erke # ;
erke% me:erke% me # ;
medvedev:medvedev # ;

Compiling

1. $ hfst-lexc test.lexc -o test.hfst
2. $ hfst-invert test.hfst | hfst-fst2fst -w -o test.hfst.ol

Testing

Some correctly analysed forms

• $ echo "erke" | hfst-proc test.hfst.ol

^erke/erke$

• $ echo "erke me" | hfst-proc test.hfst.ol

^erke me/erke me$

• $ echo "medvedev" | hfst-proc test.hfst.ol

^medvedev/medvedev$

The incorrectly analysed form

• $ echo "erke medvedev" | hfst-proc test.hfst.ol

^erke medvedev/*erke medvedev$

Expected output

This form is analysed correctly by a transducer identical to the one above except with the "erke me" form removed:

• $ echo "erke medvedev" | hfst-proc test2.hfst.ol

^erke/erke$ ^medvedev/medvedev$