Difference between revisions of "Northern Sámi and Norwegian/release"
Jump to navigation
Jump to search
(→Issues) |
|||
| (54 intermediate revisions by the same user not shown) | |||
| Line 3: | Line 3: | ||
==Issues== |
==Issues== |
||
| + | ===sme-nob-specific stuff should be in sme-nob=== |
||
| + | e.g. langs/sme/tools/mt/apertium/tagsets/apertium.nob.relabel |
||
| + | |||
| + | Alternatively (ideally), there shouldn't be sme-nob-specific tag changes; all apertium pairs should have pretty similar tagsets. |
||
| + | |||
| + | ===Periods in abbreviations missing from lemma=== |
||
| + | |||
| + | Forms "nr" and "nr." get the exact same analysis: |
||
| + | |||
| + | <pre> |
||
| + | $ echo -e 'nr\nnr.'|apertium -f none -d . sme-nob-morph |
||
| + | ^nr/nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$ |
||
| + | ^nr./nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$ |
||
| + | </pre> |
||
| + | |||
| + | So when translating to Norwegian, we have no idea whether to include the dot or not. |
||
| + | |||
| + | If form "nr." had lemma "nr." this would be simple. |
||
| + | |||
| + | |||
===High priority bad translations=== |
===High priority bad translations=== |
||
What are the high-priority linguistic issues to deal with? |
What are the high-priority linguistic issues to deal with? |
||
| − | Would we gain a lot by inserting modals instead of adverbs for |
||
| − | Pot/Cond verbs? Is there a better, general, way to translate the |
||
| − | progressive? Should we get some of Francis' [[Generating lexical selection rules|automatically discovered |
||
| − | lex.sel rules]]? And are there any "simple" constructions that we could |
||
| − | handle but don't yet? |
||
| − | * TODO: We should add a post-generator to insert epenthetics in compounds, turning eg. "ing~b" into "ingsb" |
||
* TODO: Bidix will be added to with stuff from GTSVN |
* TODO: Bidix will be added to with stuff from GTSVN |
||
| − | ** see dev/inc* (verbs need transitivity, nouns need gender checked, words with spaces shouldn't have) |
+ | ** see dev/inc* (verbs need transitivity, nouns need gender checked, words with spaces shouldn't have), see also these files with entries are not in bidix: |
| + | ** [http://apertium.svn.sourceforge.net/viewvc/apertium/incubator/apertium-sme-nob/dev/bidix-pronouns.todo.dix dev/bidix-pronouns.todo.dix] |
||
| − | * TODO: After bidix additions, Francis will run apertium-lex-learner |
||
| + | ** [http://apertium.svn.sourceforge.net/viewvc/apertium/incubator/apertium-sme-nob/dev/bidix-abbr.todo.dix dev/bidix-abbr.todo.dix] |
||
| − | * TODO: add forms to nob.dix (that are in bidix) from nn-nb-infreq; add lexicalised compounds using existing lemmas; mwe's and other missing stuff manually |
||
| + | ** [http://apertium.svn.sourceforge.net/viewvc/apertium/incubator/apertium-sme-nob/dev/bidix-shcmp.todo.dix dev/bidix-shcmp.todo.dix] |
||
| + | * TODO: After bidix additions, Francis will run apertium-lex-learner to [[Generating lexical selection rules|automatically discover |
||
| + | lex.sel rules]] |
||
* TODO: URL recognition (I tried making some with lexc, it worked by itself, but took ages to compile into the regular lexc, not sure why) |
* TODO: URL recognition (I tried making some with lexc, it worked by itself, but took ages to compile into the regular lexc, not sure why) |
||
| − | * TODO: Compounds mess up CG. We have to leave all the tags of the non-heads in because of bidix lookup, so we get eg. <code>politiija<N><Sg><Nom><Cmp>+stašuvdna<N><Sg><Ill></code> (instead of <code>politiija#stašuvdna<N><Sg><Ill></code> like the lookup2cg script gives). CG sees <code>Nom</code> and thus applies rules that it shouldn't, etc. |
||
| − | ** The ideal solution, but currently impossible: rename the non-head tags in BEFORE-/AFTER-SECTIONS. I tried <code>BEFORE-SECTIONS SUBSTITUTE (N Sg Nom Cmp) (N* Sg* Nom* Cmp)</code> which works for the above, but CG then also turns <code>duohta<A><Sg><Nom><Cmp>+dilli<N><Sg><Acc></code> into <code>duohta<A><@→N>+dilli<N*><Sg*><Nom*><Cmp><Sg><Gen>$</code>, while if there are several non-heads, it'll only substitute in the first part. It seems some sort of tag order mechanism would be needed in CG for the BEFORE-SECTIONS/AFTER-SECTIONS stashing method to work. |
||
| − | ** It's possible to do the initial renaming in a twol rule (committed, but commented out for now), but we have no way of changing the tags back in CG (an AFTER-SECTIONS rule SUBSTITUTE N* N will unfortunately merge N* and later occurences of N). |
||
| − | * DONE: Cond rules get modals, handled like passive in t1x (instead of "kanskje" in t3x), Pot gets "da<adv>" for now |
||
* DONE: In chunker/t1x, add tags to certain verbs (eg. using def-lists) that can be used in t3x to change/remove case-prepositions (see dev/valence.txt) |
* DONE: In chunker/t1x, add tags to certain verbs (eg. using def-lists) that can be used in t3x to change/remove case-prepositions (see dev/valence.txt) |
||
** TODO: fill out with more def-list entries |
** TODO: fill out with more def-list entries |
||
** TODO: go through dev/valence.txt and mark off which examples can be handled with noun-lists, and which with verb-lists |
** TODO: go through dev/valence.txt and mark off which examples can be handled with noun-lists, and which with verb-lists |
||
| + | ===Derivations mess up CG=== |
||
| − | ===DONE: remove unhandled derivations=== |
||
| + | Mainly a problem with the PoS-changing derivations. |
||
| − | Any [[Northern Sámi and Norwegian/Derivations|derivations]] that are not handled we remove from the analyser with a twol negation rule in dev/xfst2apertium.useless.twol, this makes the lexicon a lot easier to handle for testvoc. |
||
| + | In lookup2cg, PoS tags are given stars if they appear before derivational tags, so "V Der N" becomes "V* Der N". We could do this with twol, but we have no way of removing them before bidix. |
||
| − | ===DONE: +G3 tag=== |
||
| − | This is used like the +Actor tag, for sme wsd (here based on "stadieveksling"). |
||
| + | Simple / clean solution: lexicalise. |
||
| − | We move it after the +N tag in dev/xfst2apertium.hashtags.twol (so it gets the same position as +Actor), this makes transfer a lot simpler. |
||
| + | ===Testvoc=== |
||
| − | Getting a list of lemmas which have +G3 is easy (grep all nouns from bidix, analyse their lemmas and grep for lemma+G3), use this to tag bidix. |
||
| + | Before release, we need to get [[testvoc]] out of the way – making sure there are no #'s and @'s in the output. The analyser is trimmed, but we can still have generator errors (and CG substitute errors). We can't create all possible surface forms from a cyclic [[HFST]] analyser, but we can at least run as large a corpus as we can find through sme-nob and look for # and @, and also send through the right-hand-sides of the bidix. |
||
| + | There are two helper scripts: |
||
| − | ===DONE: ensure we have all necessary postchunk rules=== |
||
| − | Postchunk rules are needed for any chunk containing a |
||
| − | determiner/pronoun/adjective/noun/verb, we can easily make sure each |
||
| − | possible chunk name has a postchunk rule (new chunks are created in |
||
| − | t1x with names like pre_pre_nom, but may also be merged in t2x to |
||
| − | eg. pre_pre_nom_conj_nom) |
||
| + | * <code>dev/sme-nob.inconsistency.sh | grep '^#'</code> should give no results. This script just sends the rhs of the bidix through the generator. |
||
| − | All possible SN and SA chunks should have the needed postchunk rules now. |
||
| + | * <code>sed 's/^/sme\t/' smecorpus.txt | dev/analyse-all-stages.sh > analysedcorpus.txt</dev> – here <code>grep 'DGEN.*#'</code> should give no hits. This script also outputs the input and morph and tagger stages (and nob-sentence if handed a parallel corpus) |
||
| − | ===Testvoc=== |
||
| − | Before release, we need to get [[testvoc]] out of the way – making |
||
| − | sure there are no #'s and @'s in the output. As yet we don't have a |
||
| − | way to create all possible surface forms from an [[HFST]] analyser, |
||
| − | but we can at least run as large a corpus as we can find through |
||
| − | sme-nob and look for # and @. |
||
| − | The script dev/sme-nob.inconsistency.sh tries to generate (with nob.dix) base forms (like infinitives or singular indefinites) of the various rhs entries in the bidix. Statistics so far: |
||
| − | {|class=wikitable |
||
| − | ! Part-of-Speech !! entries from bidix that are OK in nob.dix !! in bidix but not in nob.dix !! comments !! in sme analyser but not bidix |
||
| − | |- |
||
| − | | '''verbs''' || 14017 || '''486''' || mostly mwe's missing || ??? |
||
| − | |- |
||
| − | | '''nouns''' || 3685 || '''8227''' || mostly compounds missing || ??? |
||
| − | |- |
||
| − | | '''proper nouns''' || 17412 || '''0''' || :-) easy to find nob pardefs automatically || ??? |
||
| − | |- |
||
| − | | '''adverbs''' || 96 || '''143''' || || ??? |
||
| − | |- |
||
| − | | '''prepositions''' || 42 || '''0''' || :-) || ??? |
||
| − | |- |
||
| − | | '''adjectives''' || 463 || '''593''' || || ??? |
||
| − | |- |
||
| − | | '''abbreviations''' || ??? || '''???''' || see dev/abbr.todo.dix || 619 |
||
| − | |} |
||
| − | ==Schedule== |
||
| − | {|class=wikitable |
||
| − | ! Task !! Date |
||
| − | |- |
||
| − | | Work on high priority bad translations, expand bidix coverage || until 2010-07-14 |
||
| − | |- |
||
| − | | Remove unhandled derivations, ensure we have all postchunk rules || 2010-07-14…2010-07-18 |
||
| − | |- |
||
| − | | Testvoc || 2010-07-18…2010-08-01 |
||
| − | |- |
||
| − | | '''Tentative release date for apertium-sme-nob 0.1.0''' || '''August 1st 2010''' |
||
| − | |- |
||
| − | |} |
||
| + | ====Expanding the morphology==== |
||
| − | Update: Lene and Trond have more free time after August, real release beginning of September? |
||
| + | Running <code>hfst-fst2strings sme-nob.automorf.hfst.ol</code> creates an expansion of the morphology, might be possible to use for testvoc. |
||
| + | |||
| + | There's a handy script <code>dev/gt-expand-to-bidix.sh</code> that takes as input one word and PoS (tab-separated) per line, e.g. <pre>galbmit V |
||
| + | beana N</pre>, and uses the giellatekno web-based lemma expander to create all forms of each lemma (since hfst is still not very good for that), then runs them through bidix to check for missing bidix entries. |
||
| + | |||
| + | |||
| + | |||
| + | [[Category:Northern Sámi and Norwegian|*]] |
||
Latest revision as of 13:47, 15 September 2015
This page holds information about the release schedule for apertium-sme-nob.
Contents
Issues[edit]
sme-nob-specific stuff should be in sme-nob[edit]
e.g. langs/sme/tools/mt/apertium/tagsets/apertium.nob.relabel
Alternatively (ideally), there shouldn't be sme-nob-specific tag changes; all apertium pairs should have pretty similar tagsets.
Periods in abbreviations missing from lemma[edit]
Forms "nr" and "nr." get the exact same analysis:
$ echo -e 'nr\nnr.'|apertium -f none -d . sme-nob-morph ^nr/nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$ ^nr./nr<n><abbr><acc>/nr<n><abbr><attr>/nr<n><abbr><gen>/nr<n><abbr><nom>$
So when translating to Norwegian, we have no idea whether to include the dot or not.
If form "nr." had lemma "nr." this would be simple.
High priority bad translations[edit]
What are the high-priority linguistic issues to deal with?
- TODO: Bidix will be added to with stuff from GTSVN
- see dev/inc* (verbs need transitivity, nouns need gender checked, words with spaces shouldn't have), see also these files with entries are not in bidix:
- dev/bidix-pronouns.todo.dix
- dev/bidix-abbr.todo.dix
- dev/bidix-shcmp.todo.dix
- TODO: After bidix additions, Francis will run apertium-lex-learner to automatically discover lex.sel rules
- TODO: URL recognition (I tried making some with lexc, it worked by itself, but took ages to compile into the regular lexc, not sure why)
- DONE: In chunker/t1x, add tags to certain verbs (eg. using def-lists) that can be used in t3x to change/remove case-prepositions (see dev/valence.txt)
- TODO: fill out with more def-list entries
- TODO: go through dev/valence.txt and mark off which examples can be handled with noun-lists, and which with verb-lists
Derivations mess up CG[edit]
Mainly a problem with the PoS-changing derivations.
In lookup2cg, PoS tags are given stars if they appear before derivational tags, so "V Der N" becomes "V* Der N". We could do this with twol, but we have no way of removing them before bidix.
Simple / clean solution: lexicalise.
Testvoc[edit]
Before release, we need to get testvoc out of the way – making sure there are no #'s and @'s in the output. The analyser is trimmed, but we can still have generator errors (and CG substitute errors). We can't create all possible surface forms from a cyclic HFST analyser, but we can at least run as large a corpus as we can find through sme-nob and look for # and @, and also send through the right-hand-sides of the bidix.
There are two helper scripts:
dev/sme-nob.inconsistency.sh | grep '^#'should give no results. This script just sends the rhs of the bidix through the generator.sed 's/^/sme\t/' smecorpus.txt | dev/analyse-all-stages.sh > analysedcorpus.txt</dev> – heregrep 'DGEN.*#'should give no hits. This script also outputs the input and morph and tagger stages (and nob-sentence if handed a parallel corpus)
Expanding the morphology[edit]
Running hfst-fst2strings sme-nob.automorf.hfst.ol creates an expansion of the morphology, might be possible to use for testvoc.
There's a handy script dev/gt-expand-to-bidix.sh that takes as input one word and PoS (tab-separated) per line, e.g.
galbmit V
beana N
, and uses the giellatekno web-based lemma expander to create all forms of each lemma (since hfst is still not very good for that), then runs them through bidix to check for missing bidix entries.
