User:Khannatanmai/Alternate stream modification - Revision history

Khannatanmai at 09:25, 17 July 2020

2020-07-17T09:25:15Z

Khannatanmai: /* Alternate Proposal */

2020-06-06T09:56:19Z

Alternate Proposal

Khannatanmai: /* Alternate Proposal */ Update with IDs

2020-06-06T08:15:33Z

Alternate Proposal: Update with IDs

Khannatanmai: /* Markup tags */

2020-06-01T11:37:09Z

Markup tags

Khannatanmai: /* Comparative Examples */

2020-06-01T11:35:34Z

Comparative Examples

Khannatanmai: /* IDs / Alignments */

2020-06-01T11:14:14Z

IDs / Alignments

Khannatanmai at 11:09, 1 June 2020

2020-06-01T11:09:09Z

Khannatanmai: /* Benefits */

2020-06-01T07:38:31Z

Benefits

Khannatanmai: /* Benefits */

2020-06-01T07:33:49Z

Benefits

Khannatanmai: /* Benefits */

2020-06-01T07:32:15Z

Benefits

← Older revision		Revision as of 09:25, 17 July 2020
Line 1:		Line 1:
			<strong style="color:maroon;font-size:1.5em;>The secondary tags project was shelved as the need for reading specific secondary information wasn't established. This work continued as LU-bound secondary information using [[User:Khannatanmai/Wordbound_blanks \| wordbound blanks]]. </strong>

	= [[User:Khannatanmai/New Apertium stream format \| Original Proposal]] =		= [[User:Khannatanmai/New Apertium stream format \| Original Proposal]] =

@@ Line 33: / Line 33: @@
 Update: All the secondary information will be in word-bound blanks, but each reading will have a special trailing tag which gives it a unique id in a window. Because of this, we can have reading specific information as secondary information if needed. The source reading ID will be copied over to the target reading ID automatically in the bidix lookup. Each LU will get an ID (1,2,3, etc.), and each reading inside an LU will get an id of the form token/reading (1/1, 1/2, 1/3, etc.). Compounds will get individual tags and not specific IDs (^abc<tags><!1>+xyz<tags><!2>$) because they will be broken up in pretransfer.
-By the time we reach bidix lookup, there will only be one source reading, so we can then use ids of the form token/reading for the multiple SL readings that the bidix produces. This model will ensure that throughout the pipeline we can have unique IDs for each reading.
+By the time we reach bidix lookup, there will only be one source reading, so we can then use ids of the form token/reading for the multiple TL readings that the bidix produces. This model will ensure that throughout the pipeline we can have unique IDs for each reading.
 Updated Format: <code> ^lemma<tags><!id>$[{..secondary_information..}] </code>

@@ Line 31: / Line 31: @@
 The solution is to have secondary information in word-bound blanks. These are distinguished from the usual blanks because these will be reordered during transfer whereas normal blanks would stay as they are. Of course, there is still the issue of reading-specific information, but in that regard, we can either have secondary tags for weights (which will either be redundant by the time we reach transfer, or will be shifted to the blank), or the weights can just use primary tags (it won't be ambiguous since no other tag has floating point numbers). '''All secondary information will be in word-bound blanks before the pipeline reaches transfer'''.
+Update: All the secondary information will be in word-bound blanks, but each reading will have a special trailing tag which gives it a unique id in a window. Because of this, we can have reading specific information as secondary information if needed. The source reading ID will be copied over to the target reading ID automatically in the bidix lookup. Each LU will get an ID (1,2,3, etc.), and each reading inside an LU will get an id of the form token/reading (1/1, 1/2, 1/3, etc.). Compounds will get individual tags and not specific IDs (^abc<tags><!1>+xyz<tags><!2>$) because they will be broken up in pretransfer.
-Format: <code> ^lemma<tags>$[{..secondary_information..}] </code>
+By the time we reach bidix lookup, there will only be one source reading, so we can then use ids of the form token/reading for the multiple SL readings that the bidix produces. This model will ensure that throughout the pipeline we can have unique IDs for each reading.
+Updated Format: <code> ^lemma<tags><!id>$[{..secondary_information..}] </code>
 == Objections ==
@@ Line 37: / Line 41: @@
 * '''Information Access''': It's harder to access information inside the blanks as opposed to secondary tags. But this is a practical issue, it can definitely be implemented.
-* '''No way to represent reading specific information''': Since a word bound blank attaches to an LU, it has no good way of representing reading specific information.
 * '''Philosophical''': Blanks are used to store information that is ignored during the translation of the words. It is debatable that secondary information is this kind of information, but it's not entirely incompatible.

@@ Line 69: / Line 69: @@
 '''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$ ^plan<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$</pre>
-'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:t:s:s1234}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; t:s:s1234}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:t:s:s1234}]</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234}]</pre>
 '''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}]</pre>

@@ Line 87: / Line 87: @@
 '''Word bound blanks:''' <pre>^de<pr>/of<pr>/from<pr>$[{@sf:del; @id:11}]</pre>
+== Weights ==
+'''Secondary Tags:''' <pre>^de<pr><sf:del><id:11><W:1.6787>/of<pr><sf:del><id:11><W:5.0984>/from<pr><sf:del><id:11><W:0.0065>$</pre>
+'''Word bound blanks:''' Not Reading Specific
 = Feedback =

@@ Line 82: / Line 82: @@
 '''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.01,0.05,0.67><id:1>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.97,0.50,0.23><id:2>$ ^plan<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.12,0.56,0.89><id:3>$</pre>
-'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; @emb:0.01,0.05,0.67, @id:1}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.97,0.50,0.23, @id:2}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.12,0.56,0.89, @id:3}]</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; @emb:0.01,0.05,0.67; @id:1}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; @emb:0.97,0.50,0.23; @id:2}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; @emb:0.12,0.56,0.89; @id:3}]</pre>
+'''Secondary Tags:''' <pre>^de<pr><sf:del><id:11>/of<pr><sf:del><id:11>/from<pr><sf:del><id:11>$</pre>
+'''Word bound blanks:''' <pre>^de<pr>/of<pr>/from<pr>$[{@sf:del; @id:11}]</pre>
 = Feedback =

@@ Line 58: / Line 58: @@
 == Formalism ==
 The formalism of how the secondary information will be stored in these blanks can be discussed once there has been discussion about this idea.
+= Comparative Examples =
+== Compounds & Surface Form ==
+'''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan>$ ^plan<n><sf:intrastruktuurontwikkelingsplan>$</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan$}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan}]</pre>
+== Markup tags ==
+'''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$ ^plan<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234>$</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:t:s:s1234}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; t:s:s1234}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:t:s:s1234}]</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @markup:<span class="foo" id="bar" xml:lang="ckt">}]</pre>
+== Embeddings ==
+'''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.01,0.05,0.67>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.97,0.50,0.23>$ ^plan<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.12,0.56,0.89>$</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; emb:0.01,0.05,0.67}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.97,0.50,0.23}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.12,0.56,0.89}]</pre>
+== IDs / Alignments ==
+'''Secondary Tags:''' <pre>^intrastruktuur<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.01,0.05,0.67><id:1>$ ^ontwikkelings<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.97,0.50,0.23><id:2>$ ^plan<n><sf:intrastruktuurontwikkelingsplan><t:s:s1234><emb:0.12,0.56,0.89><id:3>$</pre>
+'''Word bound blanks:''' <pre>^intrastruktuur<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234; @emb:0.01,0.05,0.67, @id:1}] ^ontwikkelings<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.97,0.50,0.23, @id:2}] ^plan<n>$[{@sf:intrastruktuurontwikkelingsplan; @t:s:s1234, @emb:0.12,0.56,0.89, @id:3}]</pre>
 = Feedback =

@@ Line 53: / Line 53: @@
 * '''Philosophically Apertium''': LUs continue to have only linguistic information, and all secondary information goes in word bound blanks. It is more consistent with Apertium's philosophy about providing information.
+* '''Compound/Contraction Problem''': <code>^abcuxyz/abc<tags>+xyz<tags>$</code> -> <code>^abc<tags><sf:abcuxyz>$ ^xyz<tags><sf:abcuxyz>$</code>. With larger compounds this adds a lot of redundant information inside Lexical Units. Having this inside blanks: <code>^abc<tags>$[{@sf:abcuxyz}] ^xyz<tags>$[{@sf:abcuxyz}]</code> leaves the LU untouched, and is better for language pair developers.
 == Formalism ==

@@ Line 48: / Line 48: @@
 * '''FST Pattern Matching''': Blanks pass through most modules, so this won't be an issue anymore.
-* '''Arbitrary amount of information''': While secondary tags have the ability to store any amount of information, they reach a limit due to the capability of tags. With blanks, there is no limit to the amount and kind of information that is stored. In fact, we can store full markup tags instead of references. (eg. <code><span class="foo" id="bar" xml:lang="ckt"></code> ) If we want to have word embeddings in the future, it wouldn't certainly be very apt to store them in secondary tags, but word bound blanks will do the job quite well.
+* '''Arbitrary amount of information''': While secondary tags have the ability to store any amount of information, they reach a limit due to the capability of tags. With blanks, there is no limit to the amount and kind of information that is stored. In fact, we can store full markup tags instead of references. If we want to have word embeddings in the future, it wouldn't certainly be very apt to store them in secondary tags, but word bound blanks will do the job quite well.
 * '''Not much regression''': Since these are bound to LUs, they give almost all the benefits of secondary tags apart from reading specific information.