Automatically trimming a monodix - Revision history

Unhammer: /* Troubleshooting */

2026-03-06T09:41:58Z

Troubleshooting

Unhammer: /* Troubleshooting */

2026-03-06T09:41:46Z

Troubleshooting

Unhammer: /* Compounds vs trimming in HFST */

2026-03-06T09:41:22Z

Compounds vs trimming in HFST

Bech: Link to French page

2014-10-07T14:37:38Z

Link to French page

Unhammer: /* Compounds vs trimming in HFST */

2014-06-20T19:28:10Z

Compounds vs trimming in HFST

Unhammer: /* Compounds vs trimming in HFST */

2014-06-20T16:43:36Z

Compounds vs trimming in HFST

Unhammer: /* Compounds vs trimming in HFST */

2014-06-20T14:49:50Z

Compounds vs trimming in HFST

Unhammer: /* Compounds vs trimming in HFST */

2014-02-28T09:20:27Z

Compounds vs trimming in HFST

Unhammer: /* Compounds vs trimming in HFST */

2014-02-28T09:19:22Z

Compounds vs trimming in HFST

Ilnar.salimzyan: /* Compounds vs trimming in HFST */ just a full stop

2014-02-11T14:27:31Z

Compounds vs trimming in HFST: just a full stop

@@ Line 181: / Line 181: @@
 </pre>
-(the above hfst trimming rules don't take into account that the bidix could be compiled to multiple sections; the above patch ensures lt-comp only compiles to one section)
+(the above hfst trimming rules don't take into account that the bidix could be compiled to multiple sections (many fst's in one file); the above patch ensures lt-comp only compiles to one section)
 ==lttoolbox==

@@ Line 181: / Line 181: @@
 </pre>
-(the above hfst trimming rules don't take into account that the bidix could be compiled to multiple sections)
+(the above hfst trimming rules don't take into account that the bidix could be compiled to multiple sections; the above patch ensures lt-comp only compiles to one section)
 ==lttoolbox==

@@ Line 149: / Line 149: @@
         hfst-concatenate -1 .deps/$(PREFIX1).autobil.nonplussed -2 .deps/$(PREFIX1).autobil.postplus.0,2 -o $@
 </pre>
+=== Troubleshooting ===
+If you see something like
+<pre>
+hfst-concatenate: Error: first input '.deps/sma-sme.single-plus.hfst' contains fewer transducers than second input '.deps/sma-sme.autobil.nonplussed'
+</pre>
+then apply this change:
+<pre>
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -160,13 +160,15 @@
+ ## Bilingual dictionaries
+ ###############################################################################
++LT_COMP_SINGLE_SECTION=LT_JOBS=false LT_MAX_SECTION_ENTRIES=0 lt-comp --no-split
++
+ $(PREFIX1).autobil.bin: $(BASENAME).$(PREFIX1).dix
+ 	apertium-validate-dictionary $<
+-	lt-comp lr $< $@
++	$(LT_COMP_SINGLE_SECTION) lr $< $@
+ $(PREFIX2).autobil.bin: $(BASENAME).$(PREFIX1).dix
+ 	apertium-validate-dictionary $<
+-	lt-comp rl $< $@
++	$(LT_COMP_SINGLE_SECTION) rl $< $@
+</pre>
+(the above hfst trimming rules don't take into account that the bidix could be compiled to multiple sections)
 ==lttoolbox==

← Older revision		Revision as of 14:37, 7 October 2014
Line 1:		Line 1:
			[[Tronquer automatiquement un dictionnaire morphologique\|En français]]

	There used to be a problem in Apertium regarding copies. When we came to make a new language pair based off some existing resource in Apertium (e.g. a [[monodix]]), we would make a copy of that resource, and then change it as we need. This was less than ideal because:		There used to be a problem in Apertium regarding copies. When we came to make a new language pair based off some existing resource in Apertium (e.g. a [[monodix]]), we would make a copy of that resource, and then change it as we need. This was less than ideal because:

@@ Line 110: / Line 110: @@
 === Compounds vs trimming in HFST ===
-The sme.lexc can't be trimmed using the above simple HFST trick, due to compounds.
+The sme.lexc needs a more complicated trimming system, due to compounds.
 Say you have '''cake n sg''', '''cake n pl''', '''beer n pl''' and '''beer n sg''' in monodix, while bidix has '''beer n''' and '''wine n'''. The HFST method without compounding is to intersect '''(cake|beer) n (sg|pl)''' with '''(beer|wine) n .*''' to get '''beer n (sg|pl)'''.
-But HFST represents compounding as a transition from the end of the singular noun to the beginning of the (noun) transducer, so a compounding HFST actually looks like
+But the sme lexicon represents compounding as a transition from the end of the singular noun to the beginning of the (noun) transducer, so a compounding HFST actually looks like
 : '''((cake|beer) n sg)*(cake|beer) n (sg|pl)'''
 The intersection of this with

@@ Line 134: / Line 134: @@
-Here's a make recipe for trimming with <nowiki><code>bidix [^+]* (+ bidix [^+]*)*</code></nowiki>:
+Here's a make recipe for trimming with <code><nowiki>bidix [^+]* (+ bidix [^+]*)*</nowiki></code>:
 <pre>
 .deps/$(PREFIX1).autobil.prefixes: $(PREFIX1).autobil.bin .deps/.d

@@ Line 110: / Line 110: @@
 === Compounds vs trimming in HFST ===
-The sme.lexc can't be trimmed using the simple HFST trick, due to compounds.
+The sme.lexc can't be trimmed using the above simple HFST trick, due to compounds.
 Say you have '''cake n sg''', '''cake n pl''', '''beer n pl''' and '''beer n sg''' in monodix, while bidix has '''beer n''' and '''wine n'''. The HFST method without compounding is to intersect '''(cake|beer) n (sg|pl)''' with '''(beer|wine) n .*''' to get '''beer n (sg|pl)'''.
@@ Line 130: / Line 130: @@
 In sme-nob, we ensured that anything that we wanted pretransfer to split would have a + in it, by using tag relabelling. So the FST would have paths like  <code>foo&lt;tag&gt;+bar&lt;othertag&gt;</code> which are split into  <code>foo&lt;tag&gt;</code> and <code>bar&lt;othertag&gt;</code>. (Unlike with JOIN, we can have several +'es in a row.)
-Perhaps a special-purpose hfst trimming tool could do what lt-trim does on seeing the plus.
 The downside is that lexc writers have to really ensure that anything that is supposed to be as two units in bidix really has a +, but they have to do that anyway!
+Here's a make recipe for trimming with <nowiki><code>bidix [^+]* (+ bidix [^+]*)*</code></nowiki>:
+<pre>
+.deps/$(PREFIX1).autobil.prefixes: $(PREFIX1).autobil.bin .deps/.d
+        lt-print $< | sed 's/ /@_SPACE_@/g' > .deps/$(PREFIX1).autobil.att
+        hfst-txt2fst -e ε <  .deps/$(PREFIX1).autobil.att > .deps/$(PREFIX1).autobil.hfst
+        hfst-project -p upper .deps/$(PREFIX1).autobil.hfst > .deps/$(PREFIX1).autobil.upper
+        echo ' [ ? - %+ ]* ' | hfst-regexp2fst > .deps/any-nonplus.hfst
+        hfst-concatenate -1 .deps/$(PREFIX1).autobil.upper -2 .deps/any-nonplus.hfst -o .deps/$(PREFIX1).autobil.nonplussed # bidix [^+]*
+        echo ' %+ ' | hfst-regexp2fst > .deps/single-plus.hfst
+        hfst-concatenate -1 .deps/single-plus.hfst -2 .deps/$(PREFIX1).autobil.nonplussed -o .deps/$(PREFIX1).autobil.postplus # + bidix [^+]*
+        hfst-repeat -f0 -t2 -i .deps/$(PREFIX1).autobil.postplus -o .deps/$(PREFIX1).autobil.postplus.0,2 # (+ bidix [^+]*){0,2} gives at most triple-compounds
+        hfst-concatenate -1 .deps/$(PREFIX1).autobil.nonplussed -2 .deps/$(PREFIX1).autobil.postplus.0,2 -o $@
+</pre>
 ==lttoolbox==

@@ Line 125: / Line 125: @@
 Lttoolbox doesn't represent compounding by extra circular transitions, but instead by a special restart symbol interpreted while analysing, so
-lt-trim is able to understand compounds by simply skipping the compound tags
+lt-trim is able to understand compounds by simply skipping the compound tags.
 ==lttoolbox==