Northern Sámi and Norwegian/smemorf - Revision history

Unhammer: /* Tagset changes */

2015-04-15T09:46:26Z

Tagset changes

Unhammer: /* Dashes */

2014-06-13T11:25:57Z

Dashes

Unhammer: /* Tagset changes */

2014-06-13T11:25:03Z

Tagset changes

Unhammer: deprecated

2014-06-13T11:24:49Z

deprecated

Unhammer: /* Description */

2014-06-13T11:24:38Z

Description

Unhammer: /* Updating the lexc when giellatekno/bidix changes */

2012-04-13T09:59:12Z

Updating the lexc when giellatekno/bidix changes

Unhammer: simpler

2012-04-13T09:58:17Z

simpler

Unhammer: /* Tagset changes */

2012-04-12T14:19:22Z

Tagset changes

Unhammer: /* Description */

2012-04-12T14:18:22Z

Description

Unhammer: /* Description */

2012-04-12T14:18:03Z

Description

@@ Line 13: / Line 13: @@
 The tagset in apertium-sme-nob is closer to Apertium's tagset, thus a bit different from the one in Giellatekno.
-* We remove Usage tags (+Use/Sub, etc), see the set ''Useless''
+* We remove Err/Orth/Usage tags (+Use/Sub, etc) –
 * We remove any derivational analyses that aren't yet handled by transfer/bidix
+** https://victorio.uit.no/langtech/trunk/langs/sme/tools/mt/apertium/filters/remove-derivation-strings-modifications.nob.regex
+** https://victorio.uit.no/langtech/trunk/langs/sme/src/filters/remove-illegal-derivation-strings.regex
 ** [[Northern Sámi and Norwegian/Derivations|More on derivations in sme-nob]]
+* We reorder some tags
-* We remove the - from split compound lemmas so that they may be looked up in bidix.
+** https://victorio.uit.no/langtech/trunk/langs/sme/src/filters/reorder-subpos-tags.sme.regex
-* We remove the #-mark between those compounds that are lexicalised/non-dynamic (this should not be necessary any longer?)
+* We change the format of tags, so +N becomes &lt;n&gt;, +Der/PassL becomes &lt;der_passl&gt;, etc.
-* We also ensure the +G3 tag occurs ''after'' the +N tag, a common upstream bug in the lexc files
+* We change certain tags
-* We change the format of tags, so +N becomes &lt;N&gt;, +Der/1 becomes &lt;Der_1&gt;, etc.
+** https://victorio.uit.no/langtech/trunk/langs/sme/tools/mt/apertium/tagsets/apertium.nob.relabel
+** https://victorio.uit.no/langtech/trunk/langs/sme/tools/mt/apertium/tagsets/apertium.postproc.relabel
+** https://victorio.uit.no/langtech/trunk/langs/sme/tools/mt/apertium/tagsets/modify-tags.nob.regex
 ==TODO==

@@ Line 34: / Line 34: @@
 ===Dashes===
-lexc handles dashes by adding them literally (like a lemma), doing that in with a bidix pardef would be very messy (also, doesn't it give issues with lemma-matching in CG?). Currently we remove the dashes in xfst2apertium.useless.twol (and in certain cases re-add them in transfer as the tag &lt;dash&gt;), but perhaps we could add a tag there …
+lexc handles dashes by adding them literally (like a lemma), doing that in with a bidix pardef would be very messy (also, doesn't it give issues with lemma-matching in CG?).
 ===Multiwords===

@@ Line 13: / Line 13: @@
 The tagset in apertium-sme-nob is closer to Apertium's tagset, thus a bit different from the one in Giellatekno.
-#* We remove Usage tags (+Use/Sub, etc), see the set ''Useless''
+* We remove Usage tags (+Use/Sub, etc), see the set ''Useless''
-#* We remove any derivational analyses that aren't yet handled by transfer/bidix
+* We remove any derivational analyses that aren't yet handled by transfer/bidix
-#** [[Northern Sámi and Norwegian/Derivations|More on derivations in sme-nob]]
+** [[Northern Sámi and Norwegian/Derivations|More on derivations in sme-nob]]
-#* We remove the - from split compound lemmas so that they may be looked up in bidix.
+* We remove the - from split compound lemmas so that they may be looked up in bidix.
-#* We remove the #-mark between those compounds that are lexicalised/non-dynamic (this should not be necessary any longer?)
+* We remove the #-mark between those compounds that are lexicalised/non-dynamic (this should not be necessary any longer?)
-#* We also ensure the +G3 tag occurs ''after'' the +N tag, a common upstream bug in the lexc files
+* We also ensure the +G3 tag occurs ''after'' the +N tag, a common upstream bug in the lexc files
-#* We change the format of tags, so +N becomes &lt;N&gt;, +Der/1 becomes &lt;Der_1&gt;, etc.
+* We change the format of tags, so +N becomes &lt;N&gt;, +Der/1 becomes &lt;Der_1&gt;, etc.
 ==TODO==

@@ Line 22: / Line 22: @@
 ==TODO==
-===regex vs twol===
-* Investigate if we can use at least some of the xfst scripts from giellatekno instead of xfst2apertium.useless.twol.
 ===Misc===
 * add entries from bidix that are missing from the analyser

@@ Line 2: / Line 2: @@
 ==Description==
-{{TOCD}}
-The sme morphological analyser is a ''trimmed'' version of the one in Giellatekno. Everything we get from Giellatekno is contained in the files
+The sme morphological analyser is a trimmed and tag-changed version of the one in Giellatekno.
+===Trimming===
-* '''apertium-sme-nob.sme.lexc''' (lexicon)
+Trimming happens using the same HFST method as in the Turkic pairs etc. [[Automatically_trimming_a_monodix#Compounds_vs_trimming_in_HFST|Compounds are not handled correctly by this method]].
-* '''apertium-sme-nob.sme.twol''' (two-level morphology)
-The twol file is a plain copy of twol-sme.txt. The lexc file is a concatenation of sme-lex.txt and the various POS-sme-lex.txt files in gt/sme/src (e.g. verb-sme-lex.txt, adj-sme-lex.txt). However, for each of those POS-files, the apertium lexc file only contains lines where the lemma exists in bidix (see the below section).
 ===Tagset changes===
-The tagset in apertium-sme-nob is, for various reasons, a bit different from the one in Giellatekno.
+The tagset in apertium-sme-nob is closer to Apertium's tagset, thus a bit different from the one in Giellatekno.
-We don't use the regex rules (gt/common/src/*.xfst) to remove Der2 and Use/Sub tags (though we may start doing this later). Tag fixes happens using a few two-level rules and similar that are composed on the analyser:
+#* We remove Usage tags (+Use/Sub, etc), see the set ''Useless''
-# '''[http://apertium.svn.sourceforge.net/viewvc/apertium/staging/apertium-sme-nob/xfst2apertium.useless.twol?revision=HEAD xfst2apertium.useless.twol]'''
+#* We remove any derivational analyses that aren't yet handled by transfer/bidix
-#* This file is composed first
-#* It removes Usage tags (+Use/Sub, etc), see the set ''Useless''
-#* It removes any derivational analyses that aren't yet handled by transfer/bidix, see the set ''UnhandledDerivation''
 #** [[Northern Sámi and Norwegian/Derivations|More on derivations in sme-nob]]
-#* It also removes the - from split compound lemmas so that they may be looked up in bidix.
+#* We remove the - from split compound lemmas so that they may be looked up in bidix.
+#* We remove the #-mark between those compounds that are lexicalised/non-dynamic (this should not be necessary any longer?)
-# '''[http://apertium.svn.sourceforge.net/viewvc/apertium/staging/apertium-sme-nob/xfst2apertium.hashtags.twol?revision=HEAD xfst2apertium.hashtags.twol]'''
+#* We also ensure the +G3 tag occurs ''after'' the +N tag, a common upstream bug in the lexc files
-#* This file is composed second
+#* We change the format of tags, so +N becomes &lt;N&gt;, +Der/1 becomes &lt;Der_1&gt;, etc.
-#* It removes the #-mark between those compounds that are lexicalised/non-dynamic (this should not be necessary any longer)
-#* It also ensures the +G3 tag occurs ''after'' the +N tag, a common error in the lexc files
-# '''[http://apertium.svn.sourceforge.net/viewvc/apertium/staging/apertium-sme-nob/xfst2apertium.relabel?revision=HEAD xfst2apertium.relabel]'''
-#* This file is used with hfst-substitute to change the format of tags, so +N becomes &lt;N&gt;, +Der/1 becomes &lt;Der_1&gt;, etc.
-===Updating the lexc when giellatekno/bidix changes===
-We keep the lexc file up to date with the bidix and the giellatekno entries with the python script '''update-morph/update-lexc.py''' and a per-user configuration file based on '''update-morph/langs.cfg.in'''. The configuration file tells which -lex.txt source files are to be plain copied, and which are to be trimmed, and any POS tags to restrict the trimming to. For trimming, it loads the compiled bidix FST ('''sme-nob.autobil.bin'''), and, for each of the lines in the files that are to be trimmed, it checks if the lemma (plus possible POS tags) is possible to analyse with the FST. So if noun-sme-lex.txt has
-<pre>
-beron GAHPIR ;
-beroštupmi:berošt UPMI ;
-beroštus#riidu:beroštus#rij'du ALBMI ;
-beroštus#vuostálasvuohta+CmpN/SgG+CmpN/DefPlGen:beroštus#vuostálasvuoh'ta LUONDU ;
-</pre>
-and the config says to append <code><N></code> when trimming nouns, it will try sending <code>^beron<N>$ ^beroštupmi<N>$ ^beroštusvuostálasvuohta<N>$</code> through sme-nob.autobil.bin, and if beron gave a match, that line will be included, if beroštupmi didn't, it'll be excluded, etc. (If the bidix actually specified <code>^beron<N><Actor>$</code>, it would still get included since it's a partial match; it's not perfect, but it saves a lot of trouble.)
-So to add new words to the lexc:
-# the word has to be in the relevant gt/sme/src/SOMETHING.lex.txt file in your copy of Giellatekno SVN
-# the word has to be in bidix ('''apertium-sme-nob.sme-nob.dix''') with a translation
-# you have to have created '''update-morph/langs.cfg'''
-#* You do this by copying <code>update-morph/langs.cfg.in</code> and editing the <code>SRC</code> line to point to where you checked out the gt/sme/src directory from Giellatekno SVN, there's a README in the update-morph/ directory with more information.
-# then you just run <code>make</code> to generate and compile the analyser (this will first ensure bidix is compiled)
-Note: if you get a warning about not finding PYTHON, do you have to re-run autogen.sh like this: <code>sh.autogen.sh PYTHON=python2.6</code>, exchanging <code>python2.6</code> for whatever python2 version you have installed.
 ==TODO==

@@ Line 42: / Line 42: @@
 # the word has to be in giellatekno's lexc
 # the word has to be in bidix ('''apertium-sme-nob.sme-nob.dix''') with a translation
+# you have to have created '''update-morph/langs.cfg'''
-# the bidix has to be compiled (<code>make sme-nob.autobil.bin</code>)
+#* You do this by copying <code>update-morph/langs.cfg.in</code> and editing the <code>SRC</code> line to point to where you checked out the gt/sme/src directory from Giellatekno SVN, there's a README in the update-morph/ directory with more information.
-# and then you can run <code>/usr/bin/python2.6 update-morph/update-lexc.py --config=update-morph/my-langs.cfg</code>
+# then you just run <code>make</code> to generate and compile the analyser (this will first ensure bidix is compiled)
-#* you create update-morph/my-langs.cfg by copying update-morph/langs.cfg.in and editing the SRC line to point to where you checked out the sme morphology from Giellatekno svn
-# and then you can run <code>make</code> to compile the analyser
+Note: if you get a warning about not finding PYTHON, do you have to re-run autogen.sh like this: <code>sh.autogen.sh PYTHON=python2.6</code>, exchanging <code>python2.6</code> for whatever python2 version you have installed.
-For simple copy-pasting, the last three steps are:
-<pre>make sme-nob.autobil.bin &&
-/usr/bin/python2.6 update-morph/update-lexc.py --config=update-morph/my-langs.cfg &&
-make</pre>
-(given that /usr/bin/python2.6 is your python2 version, and your personal copy of langs.cfg.in is stored as update-morph/my-langs.cfg)
 ==TODO==

← Older revision		Revision as of 14:18, 12 April 2012
Line 10:		Line 10:
	* '''apertium-sme-nob.sme.twol''' (two-level morphology)		* '''apertium-sme-nob.sme.twol''' (two-level morphology)

	The twol file is a plain copy of twol-sme.txt. The lexc file is a concatenation of sme-lex.txt and the various POS-sme-lex.txt files in gt/sme/src (e.g. verb-sme-lex.txt, adj-sme-lex.txt). However, for each of those POS-files, the apertium lexc file only contains lines where the lemma exists in bidix.		The twol file is a plain copy of twol-sme.txt. The lexc file is a concatenation of sme-lex.txt and the various POS-sme-lex.txt files in gt/sme/src (e.g. verb-sme-lex.txt, adj-sme-lex.txt). However, for each of those POS-files, the apertium lexc file only contains lines where the lemma exists in bidix (see the below section).