Northern Sámi and Norwegian/release - Revision history

Unhammer: /* Issues */

2015-09-15T13:47:19Z

Issues

Unhammer: /* Issues */

2014-06-20T10:16:20Z

Issues

Unhammer: /* Issues */

2014-06-14T07:48:58Z

Issues

Unhammer: /* TODO: Testvoc */

2011-11-06T10:17:40Z

TODO: Testvoc

Unhammer: /* TODO: Derivations mess up CG */

2011-10-26T10:22:00Z

TODO: Derivations mess up CG

Unhammer: /* TODO [1/2]: Multiple identical tags per reading in CG */

2011-10-26T10:21:40Z

TODO [1/2]: Multiple identical tags per reading in CG

Unhammer: /* TODO: Testvoc */

2011-10-26T10:11:08Z

TODO: Testvoc

Unhammer: remove old schedule

2011-10-26T10:05:51Z

remove old schedule

Unhammer: /* TODO: Testvoc */

2011-10-26T10:01:09Z

TODO: Testvoc

Unhammer at 13:57, 18 November 2010

2010-11-18T13:57:24Z

@@ Line 5: / Line 5: @@
 ===sme-nob-specific stuff should be in sme-nob===
 e.g. langs/sme/tools/mt/apertium/tagsets/apertium.nob.relabel
+Alternatively (ideally), there shouldn't be sme-nob-specific tag changes; all apertium pairs should have pretty similar tagsets.
 ===Periods in abbreviations missing from lemma===
@@ Line 38: / Line 41: @@
 ** TODO: go through dev/valence.txt and mark off which examples can be handled with noun-lists, and which with verb-lists
-===TODO: Derivations mess up CG===
+===Derivations mess up CG===
 Mainly a problem with the PoS-changing derivations.
 In lookup2cg, PoS tags are given stars if they appear before derivational tags, so "V Der N" becomes "V* Der N". We could do this with twol, but we have no way of removing them before bidix.
-* Simple / clean solution: lexicalise.
-* Boring / ugly solution: add stars with twol before any PoS-changing derivation tags, then instead of
-<pre><e><p><l>geavahit<s n="V"/><s n="TV"/></l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__verb"/></e></pre>
-we have
-<pre><e><p><l>geavahit</l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__tverb"/></e></pre>
-where __tverb adds the TV tag, and all PoS changing derivations use V* instead of V.
+Simple / clean solution: lexicalise.
-===TODO: Testvoc===
-Before release, we need to get [[testvoc]] out of the way – making sure there are no #'s and @'s in the output. As yet we don't have a way to create all possible surface forms from an [[HFST]] analyser, but we can at least run as large a corpus as we can find through sme-nob and look for # and @.
+===Testvoc===
+Before release, we need to get [[testvoc]] out of the way – making sure there are no #'s and @'s in the output. The analyser is trimmed, but we can still have generator errors (and CG substitute errors). We can't create all possible surface forms from a cyclic [[HFST]] analyser, but we can at least run as large a corpus as we can find through sme-nob and look for # and @, and also send through the right-hand-sides of the bidix.
+There are two helper scripts:
+* <code>dev/sme-nob.inconsistency.sh | grep '^#'</code> should give no results. This script just sends the rhs of the bidix through the generator.
+* <code>sed 's/^/sme\t/' smecorpus.txt | dev/analyse-all-stages.sh > analysedcorpus.txt</dev> – here <code>grep 'DGEN.*#'</code> should give no hits. This script also outputs the input and morph and tagger stages (and nob-sentence if handed a parallel corpus)
-===Generation report===
-The script <code>dev/generation-test -r corpus</code> translates a corpus and gives a frequency sorted list of errors (words marked with #, / or
-@). Current status: http://apertium.svn.sourceforge.net/viewvc/apertium/staging/apertium-sme-nob/dev/generation-report.txt
-===Bidix inconsistency===
-The script dev/sme-nob.inconsistency.sh tries to generate (with nob.dix) base forms (like infinitives or singular indefinites) of the various rhs entries in the bidix. Statistics so far:
-{|class=wikitable
-! Part-of-Speech !! entries from bidix that are OK in nob.dix !! in bidix but not in nob.dix !! comments !! in sme analyser but not bidix
-|-
-|  '''verbs'''  ||  2536  || '''0''' || :) || ???
-|-
-|  '''nouns'''  ||  11758  || '''0''' || :) || ???
-|-
-|  '''proper nouns'''  ||  28310  || '''15159''' || look for PPP mark in bidix, use dev/props-from-bidix-to-nob.sh || ???
-|-
-|  '''adverbs'''  ||  235  || '''0''' || :) only one nob pardef, simple to add || ???
-|-
-|  '''prepositions'''  ||  42  || '''0''' || :) || lots missing from bidix still
-|-
-|  '''adjectives'''  ||  1056  || '''0''' || :) (not sure if all forms are covered though) || ???
-|-
-|  '''abbreviations'''  ||  ???  || '''0''' || :)  || '''0'''
-|-
-|  '''sub-/conjunctions'''  ||  25  || '''0''' || :) || ???
-|-
-|  '''pronouns'''  ||  ???  || ??? ||  || '''0'''
-|-
-|  '''ShCmp'''  ||  ???  || '''0''' || compound parts, removed from analyser || '''0'''
-|-
-|  '''Numerals'''  ||  ???  || '''???''' || bidix should be OK, not 100% sure, still lots missing from generator! || '''???'''
-|}
-===Expanding the morphology===
+====Expanding the morphology====
-Running <code>hfst-fst2strings sme-nob.automorf.hfst.ol</code> creates an expansion of the morphology, might be possible to use for testvoc? TODO
+Running <code>hfst-fst2strings sme-nob.automorf.hfst.ol</code> creates an expansion of the morphology, might be possible to use for testvoc.
 There's a handy script <code>dev/gt-expand-to-bidix.sh</code> that takes as input one word and PoS (tab-separated) per line, e.g. <pre>galbmit	V

@@ Line 3: / Line 3: @@
 ==Issues==
+===sme-nob-specific stuff should be in sme-nob===
+e.g. langs/sme/tools/mt/apertium/tagsets/apertium.nob.relabel
 ===Periods in abbreviations missing from lemma===

@@ Line 3: / Line 3: @@
 ==Issues==
+===Periods in abbreviations missing from lemma===
+Forms "nr" and "nr." get the exact same analysis:
+<pre>
+$ echo -e 'nr\nnr.'|apertium -f none -d . sme-nob-morph
+^nr/nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$
+^nr./nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$
+</pre>
+So when translating to Norwegian, we have no idea whether to include the dot or not.
+If form "nr." had lemma "nr." this would be simple.
 ===High priority bad translations===
 What are the high-priority linguistic issues to deal with?
@@ Line 17: / Line 32: @@
-* DONE: Cond rules get modals, handled like passive in t1x (instead of "kanskje" in t3x), Pot gets "da<adv>" for now
 * DONE: In chunker/t1x, add tags to certain verbs (eg. using def-lists) that can be used in t3x to change/remove case-prepositions (see dev/valence.txt)
 ** TODO: fill out with more def-list entries
 ** TODO: go through dev/valence.txt and mark off which examples can be handled with noun-lists, and which with verb-lists
-* DONE: Insert epenthetics in compounds (nob.dix has n.*.sg.ind.cmp forms for all nouns, outputting the right epenthetic based on that)
-===DONE: Compounds in CG===
-Compounds messed up CG: We have to leave all the tags of the non-heads in because of bidix lookup, so we get eg. <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code>. If CG sees <code>Nom</code> it applies rules that it shouldn't, etc.
-Fix: cg-proc now ignores anything up until the last baseform, so given <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code>, the rules will only see <code>stašuvdna<N><Sg><Ill></code> (we have a special CG feature to refer to the other sub-readings)
 ===TODO: Derivations mess up CG===
@@ Line 38: / Line 46: @@
 <pre><e><p><l>geavahit</l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__tverb"/></e></pre>
 where __tverb adds the TV tag, and all PoS changing derivations use V* instead of V.
-===DONE: remove unhandled derivations===
-Any [[Northern Sámi and Norwegian/Derivations|derivations]] that are not handled we remove from the analyser with a twol negation rule in dev/xfst2apertium.useless.twol, this makes the lexicon a lot easier to handle for testvoc.
-===DONE: +G3 tag===
-This is used like the +Actor tag, for sme wsd (here based on "stadieveksling").
-We move it after the +N tag in dev/xfst2apertium.hashtags.twol (so it gets the same position as +Actor), this makes transfer a lot simpler.
-Getting a list of lemmas which have +G3 is easy (grep all nouns from bidix, analyse their lemmas and grep for lemma+G3), use this to tag bidix.
-===DONE: ensure we have all necessary postchunk rules===
-Postchunk rules are needed for any chunk containing a
-determiner/pronoun/adjective/noun/verb, we can easily make sure each
-possible chunk name has a postchunk rule (new chunks are created in
-t1x with names like pre_pre_nom, but may also be merged in t2x to
-eg. pre_pre_nom_conj_nom)
-All possible SN and SA chunks should have the needed postchunk rules now.
-===DONE: bidix pardef to handle CG changing Plc to Sur===
-sme-dis.rle can change arbitrary Plc-tagged proper nouns into Sur tags, so bidix needs a pardef that translates LR any Sur as if it were Plc.
-(might also want to split entries where sme-lemma != nob-lemma into a Plc entry, and a Sur-one where sme==nob)
 ===TODO: Testvoc===

@@ Line 64: / Line 64: @@
 ===TODO: Testvoc===
-Before release, we need to get [[testvoc]] out of the way – making
+Before release, we need to get [[testvoc]] out of the way – making sure there are no #'s and @'s in the output. As yet we don't have a way to create all possible surface forms from an [[HFST]] analyser, but we can at least run as large a corpus as we can find through sme-nob and look for # and @.
-sure there are no #'s and @'s in the output. As yet we don't have a
-way to create all possible surface forms from an [[HFST]] analyser,
-but we can at least run as large a corpus as we can find through
-sme-nob and look for # and @.
+===Generation report===
+The script <code>dev/generation-test -r corpus</code> translates a corpus and gives a frequency sorted list of errors (words marked with #, / or
+@). Current status: http://apertium.svn.sourceforge.net/viewvc/apertium/staging/apertium-sme-nob/dev/generation-report.txt
+===Bidix inconsistency===
 The script dev/sme-nob.inconsistency.sh tries to generate (with nob.dix) base forms (like infinitives or singular indefinites) of the various rhs entries in the bidix. Statistics so far:
 {|class=wikitable
@@ Line 98: / Line 99: @@
+===Expanding the morphology===
-These generation/transfer errors need to be fixed:
+Running <code>hfst-fst2strings sme-nob.automorf.hfst.ol</code> creates an expansion of the morphology, might be possible to use for testvoc? TODO
-* http://apertium.svn.sourceforge.net/viewvc/apertium/incubator/apertium-sme-nob/dev/generation-report.txt
 There's a handy script <code>dev/gt-expand-to-bidix.sh</code> that takes as input one word and PoS (tab-separated) per line, e.g. <pre>galbmit	V

@@ Line 32: / Line 32: @@
 In lookup2cg, PoS tags are given stars if they appear before derivational tags, so "V Der N" becomes "V* Der N". We could do this with twol, but we have no way of removing them before bidix.
-** Simple / clean solution: lexicalise.
+* Simple / clean solution: lexicalise.
-** Boring / ugly solution: add stars with twol before any PoS-changing derivation tags, then instead of
+* Boring / ugly solution: add stars with twol before any PoS-changing derivation tags, then instead of
 <pre><e><p><l>geavahit<s n="V"/><s n="TV"/></l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__verb"/></e></pre>
 we have

@@ Line 23: / Line 23: @@
 * DONE: Insert epenthetics in compounds (nob.dix has n.*.sg.ind.cmp forms for all nouns, outputting the right epenthetic based on that)
-===TODO [1/2]: Multiple identical tags per reading in CG===
+===DONE: Compounds in CG===
-* DONE: Compounds mess up CG. We have to leave all the tags of the non-heads in because of bidix lookup, so we get eg. <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code> (instead of <code>politiija#stašuvdna<N><Sg><Ill></code> like the lookup2cg script gives). CG sees <code>Nom</code> and thus applies rules that it shouldn't, etc.
+Compounds messed up CG: We have to leave all the tags of the non-heads in because of bidix lookup, so we get eg. <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code>. If CG sees <code>Nom</code> it applies rules that it shouldn't, etc.
-** The ideal solution, but currently impossible: rename the non-head tags in BEFORE-/AFTER-SECTIONS. I tried <code>BEFORE-SECTIONS SUBSTITUTE (N Sg Nom Cmp) (N* Sg* Nom* Cmp)</code> which works for the above, but CG then also turns <code>duohta<A><Sg><Nom><Cmp>+dilli<N><Sg><Acc></code> into <code>duohta<A><@→N>+dilli<N*><Sg*><Nom*><Cmp><Sg><Gen>$</code>, while if there are several non-heads, it'll only substitute in the first part. It seems some sort of tag order mechanism would be needed in CG for the BEFORE-SECTIONS/AFTER-SECTIONS stashing method to work.
-** It's possible to do the initial renaming in a twol rule (committed, but commented out for now), but we have no way of changing the tags back in CG (an AFTER-SECTIONS rule SUBSTITUTE N* N will unfortunately merge N* and later occurences of N).
-** Fix: cg-proc now ignores anything up until the last baseform, so given <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code>, the rules will only see <code>stašuvdna<N><Sg><Ill></code> (later we may have CG features to refer to the other sub-readings)
+Fix: cg-proc now ignores anything up until the last baseform, so given <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code>, the rules will only see <code>stašuvdna<N><Sg><Ill></code> (we have a special CG feature to refer to the other sub-readings)
-* TODO: Derivations mess up CG. In lookup2cg, PoS tags are given stars if they appear before derivational tags. We could do this with twol, but again have no way of removing them before bidix. Also, the CG sub-reading features won't help here since we can't ignore _all_ tags up until the last derivation; say we have <code>"lemma" V TV Der/n N Sg Ind</code>, lookup2cg gives <code>"lemma" V* TV Der/n N Sg Ind</code> (leaving the TV tag intact).
-** Simple solution: lexicalise.
+===TODO: Derivations mess up CG===
-** Boring solution: do the twol renaming to add stars to any PoS-before-derivation, and then move all the PoS tags into pardefs:
+Mainly a problem with the PoS-changing derivations.
-<pre>
-<pardef n="V" c="treat V* exactly like V for bidix lookup">
+In lookup2cg, PoS tags are given stars if they appear before derivational tags, so "V Der N" becomes "V* Der N". We could do this with twol, but we have no way of removing them before bidix.
-  <e>       <p><l><s n="V"/></l><r></r></p></e>
+** Simple / clean solution: lexicalise.
-  <e r="LR"><p><l><s n="V*"/></l><r></r></p></e>
+** Boring / ugly solution: add stars with twol before any PoS-changing derivation tags, then instead of
-</pardef>
+<pre><e><p><l>geavahit<s n="V"/><s n="TV"/></l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__verb"/></e></pre>
-...
+we have
-<e><p><l>gieldit<s n="TV"/></l><r>forby<s n="vblex"/><s n="pers"/></r></p><par n="V"/><par n="__verb"/></e>
+<pre><e><p><l>geavahit</l><r>bruke<s n="vblex"/><s n="pers"/></r></p><par n="__tverb"/></e></pre>
-</pre>
+where __tverb adds the TV tag, and all PoS changing derivations use V* instead of V.
 ===DONE: remove unhandled derivations===

@@ Line 99: / Line 99: @@
 |}
-These bugs in the sme-nob analyser in GTSVN need to be fixed:
-* http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=857#c3
-* http://giellatekno.uit.no/bugzilla/show_bug.cgi?id=875
-And these generation/transfer errors need to be fixed:
+These generation/transfer errors need to be fixed:
 * http://apertium.svn.sourceforge.net/viewvc/apertium/incubator/apertium-sme-nob/dev/generation-report.txt

@@ Line 110: / Line 110: @@
 beana	N</pre>, and uses the giellatekno web-based lemma expander to create all forms of each lemma (since hfst is still not very good for that), then runs them through bidix to check for missing bidix entries.
-==Schedule==
-{|class=wikitable
-! Task !! Date
-|-
-| Work on high priority bad translations, expand bidix coverage || until 2010-07-14
-|-
-| Remove unhandled derivations, ensure we have all postchunk rules || 2010-07-14…2010-07-18
-|-
-| Testvoc || 2010-07-18…2010-08-01
-|-
-| '''Tentative release date for apertium-sme-nob 0.1.0''' || '''August 1st 2010'''
-|-
-|}
-Update: Lene and Trond have more free time after August, real release beginning of September?
 [[Category:Northern Sámi and Norwegian|*]]

← Older revision		Revision as of 13:57, 18 November 2010
Line 121:		Line 121:

	Update: Lene and Trond have more free time after August, real release beginning of September?		Update: Lene and Trond have more free time after August, real release beginning of September?

			[[Category:Northern Sámi and Norwegian\|*]]