Talk:Northern Sámi and Norwegian - Revision history

Unhammer: /* Headlines vs apertium-destxt */

2010-06-08T08:51:49Z

Headlines vs apertium-destxt

Unhammer: /* Headlines vs apertium-destxt */

2010-06-08T08:48:51Z

Headlines vs apertium-destxt

Unhammer: /* Headlines vs apertium-destxt */

2010-06-08T08:41:26Z

Headlines vs apertium-destxt

Unhammer: /* Headlines vs apertium-destxt */

2010-06-08T08:26:20Z

Headlines vs apertium-destxt

Unhammer: /* [fixed] Multiple mapping tags */

2010-06-08T08:24:06Z

[fixed] Multiple mapping tags

Unhammer: /* [fixed] Multiple mapping tags */

2010-05-23T10:50:48Z

[fixed] Multiple mapping tags

Unhammer: /* Multiple mapping tags */

2010-05-23T10:41:23Z

Multiple mapping tags

Unhammer: /* Multiple mapping tags */

2010-05-23T09:40:44Z

Multiple mapping tags

Unhammer: /* Multiple mapping tags */

2010-05-21T13:56:46Z

Multiple mapping tags

Unhammer: /* Multiple mapping tags */

2010-05-21T11:50:07Z

Multiple mapping tags

@@ Line 119: / Line 119: @@
 ]</pre>
-Could we make the formatter add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.
+Could we make the formatter add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis. If we instead got eg.
+<pre>
+foo¶[][
+]bar.[
-Whether the dot is added or not, depends on the format handler; apertium-deshtml adds a dot if we see a &lt;br&gt; followed by empty lines (but curiously not if we see the correct xhtml &lt;br/&gt;). However, the fact that it's a ''dot'' that's added instead of something else, is hardcoded in <code>deformat.xsl</code>, so all formats that say "this marks an end of sentence" will use a dot as an eos-marker.
+]fie.¶[][
+]foe¶[][
+]</pre>
+we could tag ¶ as something like <code><sent><headline></code>, and CG would have a chance to expect headline language.
+Whether the dot is added or not in any particular place, depends on the format handler; apertium-deshtml adds a dot if we see a &lt;br&gt; followed by empty lines (but curiously not if we see the correct xhtml &lt;br/&gt;). However, the fact that it's a ''dot'' that's added instead of something else, is hardcoded in <code>deformat.xsl</code>, so all formats that say "this marks an end of sentence" will use a dot as an eos-marker.
 Ideally, we should be able to give an argument to <code>apertium-desfoo</code> (for all values of <code>foo</code>) that specifies the eos-marker, since different language pairs (or modes) might want different eos-markers.

@@ Line 119: / Line 119: @@
 ]</pre>
-Should we make a format to add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.
+Could we make the formatter add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.
+Whether the dot is added or not, depends on the format handler; apertium-deshtml adds a dot if we see a &lt;br&gt; followed by empty lines (but curiously not if we see the correct xhtml &lt;br/&gt;). However, the fact that it's a ''dot'' that's added instead of something else, is hardcoded in <code>deformat.xsl</code>, so all formats that say "this marks an end of sentence" will use a dot as an eos-marker.
+Ideally, we should be able to give an argument to <code>apertium-desfoo</code> (for all values of <code>foo</code>) that specifies the eos-marker, since different language pairs (or modes) might want different eos-markers.

← Older revision		Revision as of 08:41, 8 June 2010
Line 120:		Line 120:

	Should we make a format to add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.		Should we make a format to add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.

	(apertium-deshtml does not add the dot)

@@ Line 119: / Line 119: @@
 ]</pre>
-Is there a way we could make it add something else instead? Then we could call it a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.
+Should we make a format to add something else instead? Then we could tag it as a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.
+(apertium-deshtml does not add the dot)

@@ Line 101: / Line 101: @@
 </pre>
 (this is actually how things work internally in vislcg3, but before output, regular vislcg3 merges mapping tags -- we override that in cg-proc)
+===Headlines vs apertium-destxt===
+[[apertium-destxt]] adds an extra period if we have an empty line below:
+<pre>
+$ echo 'foo
+>
+> bar.
+> fie.
+>
+> foe'|apertium-destxt
+foo.[][
+]bar.[
+]fie..[][
+]foe.[][
+]</pre>
+Is there a way we could make it add something else instead? Then we could call it a headline. As it is, we get double periods at the end of lines ending with a period and followed by empty lines, which messes up CG since the rules think this means an ellipsis.

@@ Line 91: / Line 91: @@
 </pre>
+Solution: keep readings with different mapping tags separate in the first cg-proc run
-Solved by simply overriding mergeMappings.
+<pre>
+$ echo 'guovvamánu
+.' |hfst-lookup -f apertium sme-nob.automorf.hfst |cg-proc sme-nob.rlx.bin
+^guovvamánu/guovvamánnu<N><Sg><Acc><@←OBJ>$
+^17/17<Num><Sg><Nom><@HNOUN>/17<Num><Sg><Nom><@APP-N←>/17<Num><Sg><Nom><@SUBJ>/17<Num><Sg><Nom><@SPRED>$
+^./.<CLB>$
+</pre>
+(this is actually how things work internally in vislcg3, but before output, regular vislcg3 merges mapping tags -- we override that in cg-proc)

@@ Line 73: / Line 73: @@
 Idea: <code><chunk-pos part="art"/></code> would insert <code><4></code> if a member of the def-attr "art" appeared at place 4, etc., no tag if no member of "art" appears in the chunk tags. This number would have to be computed at run-time, but that's the way it is anyway with our variable; and this tag would make the transfer files a lot more explicit and cleaner (the first time I saw <code><lit-tag v="4"/></code> a little part of me died).
-===Multiple mapping tags===
+===[fixed] Multiple mapping tags===
-This is a problem:
+This was a problem:
 <pre>
 $ echo 'guovvamánu
@@ Line 91: / Line 91: @@
 </pre>
+Solved by simply overriding mergeMappings.
-Possible solutions:
-* Manually ensure we always end up with just one mapping tag in CG (eg. with an AFTER-SECTIONS rule to pick an arbitrary tag)
-** Possibly simple, but manual consistency checking is not ideal...
-* Make cg-proc discard all but the first one
-* Make cg-proc split the mappings into several readings